Category Archives: databases

Using PostgreSQL row commit IDs and live stats

Once again, it’s time to write down some notes for personal use. Let me know if what I’m jotting down here is particularly wrong.

PostgreSQL can tell you what are the commit IDs that touched a particular table. You query the system column named xmin.

prosody=> select xmin from prosody limit 5;
  xmin
--------
 212236
 770460
 770460
 967052
   1493
(5 rows)

You can also ask for the timestamp when this commit was created; however, that requires you to start tracking timestamps for each commit ID.

prosody=> select pg_xact_commit_timestamp(xmin), * from prosody;

ERROR:  could not get commit timestamp data
HINT:  Make sure the configuration parameter "track_commit_timestamp" is set.

This seems to be rather useful for built-in tracking of the modification timestamp and for etags. If the database backend is well structured, it may be possible to structure queries in such a way to quickly check when the results were last modified, and help the web frontend avoid serving and requesting the results. I don’t have a clear way to do it yet, but while I don’t think impact on the database will be significant, it may help shave off some serving bytes or requests to other backends.

Commit IDs also roll over after 32bit, so their use on a high traffic site needs to be closely considered. Then again, by the time you have over four billion writes, your caches will probably otherwise expire anyway.

Table pg_stat_activity is interesting and lets you see the transactions and even queries in flight. This’ll be slightly messy, but click ‘view raw code’ to see the original formatting, and scroll around a bit.

$ sudo -u postgres psql
psql (10.1, server 9.6.10)
Type "help" for help.

postgres=# SELECT pid, query FROM pg_stat_activity;
 pid  |                                                                                                                                                                                                              query                                                                                                                                                            
------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 8968 | SELECT o0."id", o0."token", o0."refresh_token", o0."valid_until", o0."user_id", o0."app_id", o0."inserted_at", o0."updated_at" FROM "oauth_tokens" AS o0 WHERE (o0."token" = $1)
...
...
 9869 | SELECT pid, query FROM pg_stat_activity;
(12 rows)
postgres=# SELECT * FROM pg_stat_activity;
 datid  |   datname   | pid  | usesysid | usename  | application_name | client_addr | client_hostname | client_port |         backend_start         |          xact_start           |          query_start          |         state_change          | wait_event_type | wait_event |        state        | backend_xid | backend_xmin |                                                                                                                                                                                                              query                       
--------+-------------+------+----------+----------+------------------+-------------+-----------------+-------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------+------------+---------------------+-------------+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 146762 | yyyyyyy_dev | 8968 |   146761 | yyyyyyy |                  | 127.0.0.1   |                 |       47545 | 2018-11-08 23:44:52.761448+00 |                               | 2018-11-08 23:50:33.742608+00 | 2018-11-08 23:50:40.75014+00  |                 |            | idle                |             |              | SELECT u0."id", u0."bio", u0."email", u0."name", u0."nickname", u0."password_hash", u0."following", u0."ap_id", u0."avatar", u0."local", u0."info", u0."follower_address", u0."last_refreshed_at", u0."inserted_at", u0."updated_at" FROM "users" AS u0 WHERE (u0."id" = $1)
....
  16391 | prosody     | 8833 |    16386 | prosody  |                  | 10.0.AA.AAA |                 |       51490 | 2018-11-08 23:44:30.819644+00 | 2018-11-08 23:50:28.826344+00 | 2018-11-08 23:50:28.826344+00 | 2018-11-08 23:50:28.826362+00 |                 |            | idle in transaction |             |              | BEGIN
  12409 | postgres    | 9869 |       10 | postgres | psql             |             |                 |          -1 | 2018-11-08 23:48:29.262371+00 | 2018-11-08 23:50:41.487958+00 | 2018-11-08 23:50:41.487958+00 | 2018-11-08 23:50:41.487961+00 |                 |            | active              |             |      1606549 | SELECT * FROM pg_stat_activity;
(12 rows)