1.Nginx Internals Joshua Zhu 09/19/2009 

2. Agenda  Source code layout  Key concepts and infrastructure  The event-driven architecture  HTTP request handling  Mail proxying process  Nginx module development  Misc. topics 

3. Source Code Layout  Files  $ find . -name "*\.[hc]" -print | wc –l 234  $ ls src core event http mail misc os  Lines of code  $ find . -name "*\.[hc]" -print | xargs wc -l | tail -n1 110953 total 

4. Code Organization  core/  The backbone and infrastructure  event/  The event-driven engine and modules  http/  The HTTP server and modules  mail/  The Mail proxy server and modules  misc/  C++ compatibility test and the Google perftools module  os/  OS dependent implementation files 

5. Nginx Architecture  Non-blocking  Event driven  Single threaded[*]  One master process and several worker processes  Resource efficient  Highly modular 

6.The Big Picture 

7. Agenda  Source code layout  Key concepts and infrastructure  The event-driven architecture  HTTP request handling  Mail proxying process  Nginx module development  Misc. topics 

8. Memory Pool  Avoid memory fragmentation  Avoid memory leak  Allocation and deallocation can be very fast  Lifetime and pool size  Cycle  Connection  Request 

9. Memory Pool (cont’d)  ngx_pool_t  Small blocks  Large blocks  Free chain list  Cleanup handler list  API  ngx_palloc  memory aligned  ngx_pnalloc  ngx_pcalloc 

10.Memory Pool Example (1 Chunk) 

11.Memory Pool Example (2 Chunks) 

12. Buffer Management  Buffer  Pointers  memory  start/pos/last/end  file  file_pos/file_last/file  Flags  last_buf  last_in_chain  flush  in_file  memory  … 

13. Buffer Management (cont’d)  Buffer chain  Singly-linked list of buffers  Output chain  Context  in/free/busy chains  Output filter  Chain writer  Writer context 

14. String Utilities  ngx_str_t  data  len  sizeof() - 1  Memory related  String formatting  String comparison  String search  Base64 encoding/decoding  URI escaping/unescaping  UTF-8 decoding  String-to-number conversion 

15. Data Structures  Abstract data types  Array  List  Queue  Hash table  Red black tree  Radix tree  Characteristic  Set object values after added  keep interfaces clean  Chunked memory (part)  efficient 

16. Logging  Error log  Level  Debug  Access log  Multiple logs  Log format  variables  Per location  Rotation 

17. Configuration File  Directive  name  type  set  conf  offset  post  Parsing  ngx_conf_parse  Values  init  merge 

18. Configuration File (cont’d)  Block  events  http  server  upstream  location  if  Variables  Buildins  Other types  http_  sent_http_  upstream_http_  cookie_  arg_ 

19. Agenda  Source code layout  Key concepts and infrastructure  The event-driven architecture  HTTP request handling  Mail proxying process  Nginx module development  Misc. topics 

20. Master and Workers  Master  Monitor workers, respawn when a worker dies  Handle signals and notify workers  exit  reconfiguration  update  log rotation  …  Worker  Process client requests  handle connections  Get cmd from master 

21.Master Process Cycle 

22.Worker Process Cycle 

23.Inter-process Communication  Signals  Channel  socketpair  command  Shared memory  Connection counter  Stat  Atomic & spinlock  Mutex 

24. Event  ngx_event_t  Read  Write  Timeout  Callbacks  Handlers  ngx_event_accept  ngx_process_events_and_timers  ngx_handle_read_event  ngx_handle_write_event  Posted events  Posted accept events queue  Posted events queue 

25. Time Cache  The overhead of gettimeofday()  Time cache variables  ngx_cached_time  ngx_current_msec  Time strings  ngx_cached_err_log_time  ngx_cached_http_time  ngx_cached_http_log_time  Timer resolution  Interval timer  setitimer() 

26.Events and Timers Processing 

27. Timer Management  Actions  Add a timer  Delete a timer  Get the minimum timer  Red black tree[*]  O(log n) complexity 

28. Accept Mutex  Thundering herd  Serialize accept()  Lock/unlock  Listening sockets  Delay 

29. I/O  Multiplexing  kqueue/epoll  NGX_USE_CLEAR_EVENT (edge triggered)  select/poll/dev/poll  NGX_USE_LEVEL_EVENT (level triggered)  …  Advanced I/O  sendfile()  writev()  direct I/O  mmap()  AIO  TCP/IP options  TCP_CORK/TCP_NODELAY/TCP_DEFER_ACCEPT