{"id":438,"date":"2023-09-24T21:00:09","date_gmt":"2023-09-24T18:00:09","guid":{"rendered":"http:\/\/pgdataeraold.local\/?p=438"},"modified":"2023-09-24T21:00:09","modified_gmt":"2023-09-24T18:00:09","slug":"process-architecture","status":"publish","type":"post","link":"https:\/\/pgdataera.com\/en\/process-architecture\/","title":{"rendered":"Process Architecture"},"content":{"rendered":"
\n
\"\"<\/figure><\/div>\n\n\n

We can divide the processes into 4 main groups.<\/p>\n\n\n\n

    \n
  1. Postgres Process (Formerly Postmaster)<\/li>\n\n\n\n
  2. Background Process<\/li>\n\n\n\n
  3. Backend (server) Process<\/li>\n\n\n\n
  4. User\/Client Process<\/li>\n<\/ol>\n\n\n\n

    Note: The Stats collector process is included in the core system with version 15, it is not
    a separate process.<\/mark><\/p>\n\n\n\n

    Postgres Process<\/h2>\n\n\n\n
    ps -ef | grep postgres\n\npostgres 2450 1 \/usr\/lib\/postgresql\/14\/bin\/postgres -D\n\/var\/lib\/postgresql\/14\/main -c config_file=\/etc\/postgresql\/14\/main\/postgresql.conf\npostgres 2451 2450 14\/main: logger\npostgres 2453 2450 14\/main: checkpointer\npostgres 2454 2450 14\/main: background writer\npostgres 2455 2450 14\/main: walwriter\npostgres 2456 2450 14\/main: stats collector\npostgres 2457 2450 14\/main: pg_cron launcher\npostgres 2458 2450 14\/main: logical replication launcher\n\nll \/usr\/lib\/postgresql\/14\/bin\/postmaster\n\/usr\/lib\/postgresql\/14\/bin\/postmaster --> postgres<\/pre>\n\n\n\n
    Postgres process in Debian-based Linux (path is different in RHEL-based Linux.)<\/p>\n\n\n\n
    It is the parent process that first opens in the instance and starts other processes. As
    can be seen in the example, the parent process ID is 2450. Other Postgres processes
    have the parent ID as 2450.<\/p>\n\n\n\n
    Postgres process;<\/strong><\/p>\n\n\n\n
      \n
    • It takes the memory from the operating system.<\/li>\n\n\n\n
    • Starts background processes.<\/li>\n\n\n\n
    • If necessary, it performs the recovery process.<\/li>\n\n\n\n
    • It listens to the network and accepts the connection requests to the Postgresql
      database (it acts as a listener).<\/li>\n\n\n\n
    • By default, it listens to TCP 5432 port, it can be changed (need to stop and
      start the instance).<\/li>\n\n\n\n
    • Writes operation or error messages during boot to the log file (\/var\/log\/postgresql\/postgresql-\u2026..log).<\/li>\n\n\n\n
    • A postgres process can manage one cluster (the concept of cluster is explained
      in the Storage section)<\/li>\n<\/ul>\n\n\n\n
      Background (Utility) Processes<\/h2>\n\n\n\n
      pgrep -a postgres\n2450 \/usr\/lib\/postgresql\/14\/bin\/postgres -D \/var\/lib\/\npostgresql\/14\/main -c config_file=\/etc\/postgresql\/14\/main\/\npostgresql.conf\n2451 postgres: 14\/main: logger\n2453 postgres: 14\/main: checkpointer\n2454 postgres: 14\/main: background writer\n2455 postgres: 14\/main: walwriter\n2456 postgres: 14\/main: stats collector\n2457 postgres: 14\/main: pg_cron launcher\n2458 postgres: 14\/main: logical replication launcher\n\npstree -p 2450\npostgres(2450)\u2500\u252c\u2500postgres(2451)\n\u251c\u2500postgres(2453)\n\u251c\u2500postgres(2454)\n\u251c\u2500postgres(2455)\n\u251c\u2500postgres(2456)\n\u251c\u2500postgres(2457)\n\u2514\u2500postgres(2458)<\/pre>\n\n\n\n
      When we examine the ID of the parent (main) postgres process with pstree,
      background and backend\/server processes are seen.
      These are the processes required for the instance to work.
      They are initialized when the instance is started.
      Some of them starts working with the activated feature.
      BG Writer, Logger, Checkpointer, WAL writer, Autovacuum launcher, Archiver, Stats
      collector, WAL sender\/receiver etc<\/p>\n\n\n\n
      Backgorund Writer Proses<\/strong><\/h2>\n\n\n
      \n
      \"\"<\/figure><\/div>\n\n\n
      \n\n\n\n
      When the space in the shared buffer is insufficient, it writes the rarely used dirty buffer
      areas to the disk and makes these areas available again.
      Writes asynchronously.
      When it is not writing (idle), it waits in sleep mode, it does not close itself.
      How long it can stay in sleep mode can be set with the bgwriter_delay<\/code> parameter.
      It runs every 200ms, which is the default value.<\/p>\n\n\n\n
      bgwriter_lru_maxpages<\/strong><\/code>: It determines how many blocks (pages) will be
      written in each run.
      bgwriter_lru_multiplier:<\/strong><\/code> It is used to reach the required clean buffer
      amount.
      (When the multiplier value is 2, and if 50 blocks of clean buffer space is required, 100
      blocks of dirty shared buffer space are written to disk and marked as clean buffer.)<\/p>\n\n\n\n
      Dirty Buffer: New or processed data blocks in memory.
      Clean Buffer: Available memory space.<\/p>\n\n\n\n
      Checkpoint Proses<\/h2>\n\n\n
      \n
      \"\"<\/figure><\/div>\n\n\n
      \n\n\n\n
      Checkpoint locates address where the recovery process will start (transaction
      sequence).
      At checkpoint time, all data blocks in dirty buffer<\/strong><\/code> are written to disk and special records
      of the checkpoint are written to the log (replorigin) file. (The metadata information of
      the changes made in the database beforehand is written to the WAL\/REDO files.)<\/p>\n\n\n\n
      At the same time, transaction information recorded on the disk is entered into the WAL
      file. And the blocks in the shared buffer are marked as clean so they can be reused.<\/p>\n\n\n\n
      For example, during electrical\/hardware failure etc. The physical shutdown of the
      server causes the database to become inconsistent. In such cases, recovery starts
      while the database is opened.<\/mark> <\/p>\n\n\n\n
      The recovery process receives the latest checkpoint
      information and re-runs the transactions that could not be written to the disk from the
      WAL files, making the data files consistent and starts the instance.<\/p>\n\n\n\n
      When the checkpoint occurs, it will write all the data blocks in the dirty buffer<\/strong><\/code> to disk,
      which will cause I\/O load. I\/O starts from the checkpoint start point and completes
      before the next checkpoint, therefore, the checkpoint process is limited.<\/p>\n\n\n\n
      The frequency of checkpoint operation is determined by the checkpoint_timeout<\/strong><\/code>
      parameter. The default value is 5 minutes. So, the checkpointer starts the checkpoint
      process every 5 minutes or at max_wal_size<\/strong><\/code>, whichever is reached first.
      If WAL has not been written since the previous checkpoint, that is, there is no
      transaction, the checkpoint is not performed even if the checkpoint_timeout<\/strong><\/code> period
      has passed.<\/p>\n\n\n\n
      If the checkpoint_timeout<\/strong><\/code> and\/or max_wal_size<\/strong><\/code> values are reduced,
      frequent checkpoints will occur, but the recovery time will be shortened. Since
      frequent checkpoints will be costly, it should be kept in balance. The database needs to
      be adjusted according to its intended use and load.
      To identify if checkpoint is happening too frequently, the checkpoint_warning<\/strong><\/code>
      parameter can be set, which will record the warning messages in the log file.<\/p>\n\n\n\n
      WAL Writer Proses<\/h2>\n\n\n
      \n
      \"\"<\/figure><\/div>\n\n\n
      \n\n\n\n
      The Checkpointer process frequently performs the checkpoint operation. Changes in
      data are written to disk asynchronously.
      Modified data blocks are kept in the shared buffer.
      The metadata records of the change are kept in the WAL buffer.
      When the commit happens, the REDO records in the WAL buffer are written to the WAL
      segments.<\/p>\n\n\n\n
      wal_level:<\/strong> Defines which records WAL files contain.
      The default value is replica. This value is sufficient for WAL archiving, and hot standby
      (read only).
      In minimal, it deletes all logs except the information required in crash and immediate
      shutdown situations. Not suitable for base backup. The database does not contain
      the necessary information for base backup restore and recovery operations. For base
      backup, it should be a replica or logical.
      In addition to replica in logical, it also contains decoding information required for
      logical standby.<\/p>\n\n\n\n
      wal_writer_delay:<\/strong> Determines how often the wal buffer is flushed and its
      contents are written to disk.<\/p>\n\n\n\n
      wal_keep_size:<\/strong> Determines the minimum WAL size that should be kept in the
      pg_wal<\/code> directory. If your standby servers lag behind for any reason, it updates itself
      with the WAL files in pg_wal.<\/p>\n\n\n\n
      min_wal_size:<\/strong> As long as the disk usage of WAL files is smaller than this size, it
      will be overwritten and reused in checkpoint operations instead of deleting old WAL
      files. Thus, the disk space specified here is reserved. The default value is 80 MB.<\/p>\n\n\n\n
      max_wal_size:<\/strong> The size that is allowed to grow during the automatic checkpoint
      process. It is the soft limit, under heavy load, when archive_command<\/code> receives an error
      and cannot archive, or when wal_keep_size<\/code> is given a high value, it can exceed the
      size specified here.<\/p>\n\n\n\n
       <\/p>\n\n\n\n
      WAL Archiver Proses<\/h2>\n\n\n
      \n
      \"\"<\/figure><\/div>\n\n\n
      \n\n\n\n
      The WAL archiver process archives WAL files to keep backup and standby servers up
      to date. It is disabled by default. It is an optional parameter. It is activated with the
      archive_mode<\/strong><\/code> parameter. It has two modes; \u2018on<\/code>\u2019 and \u2018always<\/code>\u2019.
      There is no difference in normal operation. But if \u2018always<\/code>\u2019 is selected, it will continue to
      archive even in recovery and standby mode.
      wal_level<\/code> determines how much details are written to WA files.
      Its minimal value is not sufficient for archiving, so it cannot be enabled.<\/p>\n\n\n\n
      archive_command = 'rsync -av %p \/RA\/archive\/<\/code><\/mark>'<\/mark><\/kbd>
      Pgbackrest and other backup tools take different values according to the environments used.                                                             Example: archive_command = 'pgbackrest --stanza=mx_stanza archive-push %p'<\/mark><\/code><\/p>\n\n\n\n
      Note:<\/mark><\/p>\n\n\n\n
      1. Because online backup and standby servers are essential for critical databases,
      enabling WAL archiving is almost mandatory.<\/mark><\/p>\n\n\n\n
      2.The database activities like a dump, restore, batch, etc. can be considered for these
      settings. Stopping archiving before heavy WAL-generating operations will be beneficial
      for performance.
      (archive_mode off, wal_level = minimal and max_wal_senders =0<\/code>). It needs instance restart after the setting.<\/mark><\/p>\n\n\n\n
      Logger Proses<\/h2>\n\n\n\n
      It is the process that writes the events in the database to the current log file.
      Logging collector is turned off by default and only stderr messages are directed to the
      system\u2019s default log directory when it is turned off.
      (\/var\/log\/postgresql\/postgresql-14-main.log<\/code>)
      When enabled with logging_collector = on<\/strong><\/code> (restart required), the logger
      process opens and starts writing log files to the directory specified in the log_directory<\/code>
      parameter.<\/p>\n\n\n\n
      For example: \/var\/lib\/postgresql\/14\/main\/log\/postgresql-09-23.log<\/code>
      The full path can also be given (the postgres user must have write access to the folder).
      If path is not given, it is recorded by default to the location \/var\/log\/postgresql.<\/strong><\/kbd><\/p>\n\n\n\n
      Autovacuum Launcher Proses<\/h2>\n\n\n
      \n
      \"\"<\/figure><\/div>\n\n\n
      \n\n\n\n
      It is an optional process. It is enabled by default (autovacuum=on<\/strong><\/code>).
      Autovacuum worker processes make idle(dead) areas in datafiles usable again.
      Autovacuum launcher runs these worker processes.<\/p>\n\n\n\n
      Vacuum<\/h2>\n\n\n\n
      When a row in a table is deleted, Postgresql does not immediately delete these lines
      from the data file.
      These lines are marked as deleted.
      Likewise, updating a row roughly corresponds to a delete and an insert operation.
      The state of the row before the update is still in the data file. As a result, idle(dead)
      areas that cannot be used in data files are created. These idle fields are called dead
      records.
      The vacuum process marks dead records as reusable.
      The vacuum process does not lock the table.
      The vacuum command can be run manually<\/p>\n\n\n\n
      Vacuum Full<\/h2>\n\n\n\n
      It deletes the records marked as reusable by the Vacuum process, reorganizes the
      table, and restores the idle areas to the system.
      Vacuum full creates an empty data file.
      It copies the non-dead record <\/code>to the new datafile and empties the original file.
      Note, vacuum full operation locks the table in exclusive mode. It should be preferred
      when the transaction is less. If necessary, it should be stopped.
      There must be enough space on the disk as it will create a copy of the table.<\/p>\n\n\n\n
      Vacuum analyze<\/h2>\n\n\n\n
      It reads the number of records in the tables and generates statistics for the query
      planner.
      The Autovacuum launcher process also automates these vacuum commands.<\/p>\n\n\n\n
      Stats Collector Proses<\/h2>\n\n\n
      \n
      \"\"<\/figure><\/div>\n\n\n
      \n\n\n\n
      It is an optional process, and enabled by default.
      It is a subsystem that collects and reports activity information in the database.
      It can aggregate the number of accesses to tables and indexes at the disk block and row
      level.
      It can check the number of rows in each table, do analysis and vacuum operations.
      It can track the number of times user-defined functions are called\/executed and how
      long they take.<\/p>\n\n\n\n
      Postgresql can report all the instantaneous activities in the database.
      For example<\/strong>, commands, connections, and sessions run by server processes. This
      property is independent of the collector process.<\/mark><\/p>\n\n\n\n
      These statistics are kept in the directory listed in the stats_temp_directory<\/code>
      parameter. In databases operating under heavy load, keeping this directory on fast
      disks such as RAM-based disk\/SSD will increase performance. When the Instance is
      stopped, a permanent copy of the statistics is copied to the $PGDATA\/pg_stat_tmp<\/code> directory. Thus, stats are not lost when it is stopped and started.<\/p>\n\n\n\n
      If the instance is stopped inconsistently (immediate shutdown, server crash), statistics
      will be lost as it will be started with recovery. Likewise, it is lost in the point-in-time
      recovery process. (With version 15, the core is integrated into the system, it no longer
      works as a separate process.)<\/p>\n\n\n\n
      #Activate\/deactivate statistics;\nalter system set track_counts='on<\/strong>';<\/strong>\n\n#Statistics of commands run by session \/ server processes\nalter system set track_activities='on\/off';<\/strong>\n\n#User defined functions\nalter system set track_functions = 'none, pl, all';<\/strong>\n\n# I\/O monitoring\nalter system set track_io_timing= 'on\/off';<\/strong>\n\n#Default de\u011ferlere geri d\u00f6nmek i\u00e7in;\nalter system reset track_activities;\nalter system reset track_counts;\nalter system reset track_functions;\nalter system reset track_io_timing;<\/strong><\/code><\/pre>\n\n\n\n
      \n\n\n\n
      Gathering statistics will add overhead to the database. When the detailed analysis is
      required, statistics can be enabled and then disabled.
      track_counts: <\/strong>Determines whether to collect activity statistics in the database,
      and statistics about indexes and tables. It is enabled by default because autovacuum
      needs this information.
      track_activities:<\/strong> It controls the monitoring of the commands currently
      executed by the server process. It is enabled by default. Superuser can make changes.
      The superuser and session owner can query the reports here. It does not create a
      security vulnerability.
      track_functions: <\/strong>Enables tracking of user-defined functions.
      track_io_timing:<\/strong> Enables monitoring of block read and write operations.<\/p>\n\n\n\n
      Statistics System Views<\/h2>\n\n\n\n
      postgres=# \\d pg_stat<\/strong>\npg_stat_activity              pg_statistic\npg_stat_all_indexes           pg_statistic_ext\npg_stat_all_tables            pg_statistic_ext_data\npg_stat_archiver              pg_statistic_ext_data_stxoid_index\npg_stat_bgwriter              pg_statistic_ext_name_index\npg_stat_database              pg_statistic_ext_oid_index\npg_stat_database_conflicts    pg_statistic_ext_relid_index\npg_stat_gssapi                pg_statistic_relid_att_inh_index\npg_statio_all_indexes         pg_stat_progress_analyze\n....<\/code><\/pre>\n\n\n\n
      Statistics collected from dynamic statistics views can be queried.
      If we connect with psql<\/code> and type  \\pg_stat<\/code>  and press tab twice, the above list
      will be displayed.
      Note:<\/strong> By examining this statistical information, performance improvement can be
      made by moving tables or tablespaces with a lot of I\/O operations to the fast disks.<\/mark><\/p>\n\n\n\n
      WAL Sender\/Receiver<\/h2>\n\n\n
      \n
      \"\"<\/figure><\/div>\n\n\n
      \n\n\n\n
      These processes perform replication in active\/passive cluster architectures. The WAL
      receiver process runs on the slave(standby) server(node) and connects to the master
      (primary) server(node) via TCP\/IP.<\/p>\n\n\n\n
      The master node has the WAL sender process. WAL sender is responsible for sending
      WAL files to slave node. The WAL receiver is also responsible for receiving these WAL
      files.<\/p>\n\n\n\n
      Backend (Server) Proses<\/h2>\n\n\n
      \n
      \"\"<\/figure><\/div>\n\n\n
      \n\n\n\n
      Each session corresponds to a backend\/server process.
      It is the process that executes users\u2019 requests (select, insert, update<\/strong><\/code> etc.)
      on the Postgresql side.
      Each session takes its own backend memory space from the system for its use.
      Maximum number of backend\/server processes is limited by the max_connections<\/strong><\/code>
      parameter.
      The default value is 100.<\/p>\n\n\n\n
      Client\/User Proses<\/h2>\n\n\n\n
      A program or tool that connects to a database server or network.
      Example: Pgadmin, psql web application, java application etc.<\/mark><\/p>\n","protected":false},"excerpt":{"rendered":"
      We can divide the processes into 4 main groups. Note: The Stats collector process is included in the core system with version 15, it is nota separate process. Postgres Process Postgres process in Debian-based Linux (path is different in RHEL-based Linux.) It is the parent process that first opens in the instance and starts other …  Process Architecture<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[277],"tags":[],"class_list":["post-438","post","type-post","status-publish","format-standard","hentry","category-postgresql-database-architecture"],"_links":{"self":[{"href":"https:\/\/pgdataera.com\/en\/wp-json\/wp\/v2\/posts\/438","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pgdataera.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/pgdataera.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/pgdataera.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/pgdataera.com\/en\/wp-json\/wp\/v2\/comments?post=438"}],"version-history":[{"count":0,"href":"https:\/\/pgdataera.com\/en\/wp-json\/wp\/v2\/posts\/438\/revisions"}],"wp:attachment":[{"href":"https:\/\/pgdataera.com\/en\/wp-json\/wp\/v2\/media?parent=438"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/pgdataera.com\/en\/wp-json\/wp\/v2\/categories?post=438"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/pgdataera.com\/en\/wp-json\/wp\/v2\/tags?post=438"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}