postgres borrows concepts from mysql (now mariadb)
postgres and mariadb are borth great databases, thanks all involved.
(how to setup postgres 12 on centos 8 (very very latest))
let’s dive into hit: how to postgres:
hostnamectl; # tested on Virtualization: kvm Operating System: CentOS Linux 8 (Core) CPE OS Name: cpe:/o:centos:centos:8 Kernel: Linux 4.18.0-147.3.1.el8_1.x86_64 Architecture: x86-64 psql --version; # what version is installed? psql (PostgreSQL) 12.1 # view main config file vim /var/lib/pgsql/12/data/postgresql.conf # status systemctl status postgresql-12.service # example output ● postgresql-12.service - PostgreSQL 12 database server Loaded: loaded (/usr/lib/systemd/system/postgresql-12.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2020-05-14 11:22:29 CEST; 1 weeks 4 days ago Docs: https://www.postgresql.org/docs/12/static/ Process: 1199 ExecStartPre=/usr/pgsql-12/bin/postgresql-12-check-db-dir ${PGDATA} (code=exited, status=0/SUCCESS) Main PID: 1226 (postmaster) Tasks: 9 (limit: 26213) Memory: 423.1M CGroup: /system.slice/postgresql-12.service ├─ 1226 /usr/pgsql-12/bin/postmaster -D /var/lib/pgsql/12/data/ ├─ 1248 postgres: logger ├─ 1252 postgres: checkpointer ├─ 1253 postgres: background writer ├─ 1254 postgres: walwriter ├─ 1255 postgres: autovacuum launcher ├─ 1256 postgres: stats collector ├─ 1257 postgres: logical replication launcher └─19273 postgres: dbuser dbname 127.0.0.1(59972) idle # restart systemctl restart postgresql-12 # become user postgres su - postgres # change/reset password of user postgres psql -c "alter user postgres with password 'StrongPassword'" su - postgres -c "psql -c \"alter user postgres with password 'postgres'\"" # login as user postgres # "postgres" user in postgres # is like "root" in linux psql -U postgres # login and select dbname psql -d dbname -U dbuser # login to a specific host with ip 127.0.0.1 psql -U dbuser -h 127.0.0.1 dbname # if user gets error: psql: error: could not connect to server: FATAL: Ident authentication failed for user # edit this file vim /var/lib/pgsql/12/data/pg_hba.conf # TYPE DATABASE USER ADDRESS METHOD # "local" is for Unix domain socket connections only local all all md5 # IPv4 local connections: host all all 127.0.0.1/32 md5 # IPv6 local connections: host all all ::1/128 md5 # hit ESC # then :wq # write and quit in vim # restart postgres and try login again systemctl restart postgresql-12 # try login as postgres root user psql -U postgres Password for user postgres: psql (12.1) Type "help" for help. postgres=# <- YESSSS! LOGGED IN :) # list all databases \l List of databases Name | Owner | Encoding | Collate | Ctype | Access privileges -----------+----------+----------+-------------+-------------+----------------------- dbname | dbuser | UTF8 | en_US.UTF-8 | en_US.UTF-8 | postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres + | | | | | postgres=CTc/postgres template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres + | | | | | postgres=CTc/postgres (4 rows) # create new user CREATE USER dbuser WITH PASSWORD 'dbname'; # create new database CREATE DATABASE dbname ENCODING 'UTF-8' TEMPLATE template0 OWNER dbuser; # display short help \h # display extensive help \? General \copyright show PostgreSQL usage and distribution terms \crosstabview [COLUMNS] execute query and display results in crosstab \errverbose show most recent error message at maximum verbosity \g [FILE] or ; execute query (and send results to file or |pipe) \gdesc describe result of query, without executing it \gexec execute query, then execute each value in its result \gset [PREFIX] execute query and store results in psql variables \gx [FILE] as \g, but forces expanded output mode \q quit psql \watch [SEC] execute query every SEC seconds Help \? [commands] show help on backslash commands \? options show help on psql command-line options \? variables show help on special variables \h [NAME] help on syntax of SQL commands, * for all commands Query Buffer \e [FILE] [LINE] edit the query buffer (or file) with external editor \ef [FUNCNAME [LINE]] edit function definition with external editor \ev [VIEWNAME [LINE]] edit view definition with external editor \p show the contents of the query buffer \r reset (clear) the query buffer \s [FILE] display history or save it to file \w FILE write query buffer to file Input/Output \copy ... perform SQL COPY with data stream to the client host \echo [STRING] write string to standard output \i FILE execute commands from file \ir FILE as \i, but relative to location of current script \o [FILE] send all query results to file or |pipe \qecho [STRING] write string to query output stream (see \o) Conditional \if EXPR begin conditional block \elif EXPR alternative within current conditional block \else final alternative within current conditional block \endif end conditional block Informational (options: S = show system objects, + = additional detail) \d[S+] list tables, views, and sequences \d[S+] NAME describe table, view, sequence, or index \da[S] [PATTERN] list aggregates \dA[+] [PATTERN] list access methods \db[+] [PATTERN] list tablespaces \dc[S+] [PATTERN] list conversions \dC[+] [PATTERN] list casts \dd[S] [PATTERN] show object descriptions not displayed elsewhere \dD[S+] [PATTERN] list domains \ddp [PATTERN] list default privileges \dE[S+] [PATTERN] list foreign tables \det[+] [PATTERN] list foreign tables \des[+] [PATTERN] list foreign servers \deu[+] [PATTERN] list user mappings \dew[+] [PATTERN] list foreign-data wrappers \df[anptw][S+] [PATRN] list [only agg/normal/procedures/trigger/window] functions \dF[+] [PATTERN] list text search configurations \dFd[+] [PATTERN] list text search dictionaries \dFp[+] [PATTERN] list text search parsers \dFt[+] [PATTERN] list text search templates \dg[S+] [PATTERN] list roles \di[S+] [PATTERN] list indexes \dl list large objects, same as \lo_list \dL[S+] [PATTERN] list procedural languages \dm[S+] [PATTERN] list materialized views \dn[S+] [PATTERN] list schemas \do[S] [PATTERN] list operators \dO[S+] [PATTERN] list collations \dp [PATTERN] list table, view, and sequence access privileges \dP[itn+] [PATTERN] list [only index/table] partitioned relations [n=nested] \drds [PATRN1 [PATRN2]] list per-database role settings \dRp[+] [PATTERN] list replication publications \dRs[+] [PATTERN] list replication subscriptions \ds[S+] [PATTERN] list sequences \dt[S+] [PATTERN] list tables \dT[S+] [PATTERN] list data types \du[S+] [PATTERN] list roles \dv[S+] [PATTERN] list views \dx[+] [PATTERN] list extensions \dy [PATTERN] list event triggers \l[+] [PATTERN] list databases \sf[+] FUNCNAME show a function's definition \sv[+] VIEWNAME show a view's definition \z [PATTERN] same as \dp Formatting \a toggle between unaligned and aligned output mode \C [STRING] set table title, or unset if none \f [STRING] show or set field separator for unaligned query output \H toggle HTML output mode (currently off) \pset [NAME [VALUE]] set table output option (border|columns|csv_fieldsep|expanded|fieldsep| fieldsep_zero|footer|format|linestyle|null| numericlocale|pager|pager_min_lines|recordsep| recordsep_zero|tableattr|title|tuples_only| unicode_border_linestyle|unicode_column_linestyle| unicode_header_linestyle) \t [on|off] show only rows (currently off) \T [STRING] set HTML <table> tag attributes, or unset if none \x [on|off|auto] toggle expanded output (currently off) Connection \c[onnect] {[DBNAME|- USER|- HOST|- PORT|-] | conninfo} connect to new database (currently "cyclos4") \conninfo display information about current connection \encoding [ENCODING] show or set client encoding \password [USERNAME] securely change the password for a user Operating System \cd [DIR] change the current working directory \setenv NAME [VALUE] set or unset environment variable \timing [on|off] toggle timing of commands (currently off) \! [COMMAND] execute command in shell or start interactive shell Variables \prompt [TEXT] NAME prompt user to set internal variable \set [NAME [VALUE]] set internal variable, or list all if no parameters \unset NAME unset (delete) internal variable Large Objects \lo_export LOBOID FILE \lo_import FILE [COMMENT] \lo_list \lo_unlink LOBOID large object operations
# use Ctrl+D # or type \q # to quit/logoff postgres # backup all databases to one file pg_dumpall > $(date '+%Y-%m-%d-%H').backup.postgre.sql # also compress it in one go time pg_dumpall | gzip > $(date '+%Y-%m-%d-%H').backup_all_postgre_databases.sql.gz # or if root is not allowed # try with postgres default user "postgres" pg_dumpall -U postgres > $(date '+%Y-%m-%d-%H').backup.postgresql # restore all databases (src) psql -f 2020-08-10-09.backup.postgresql postgres # or copy the backup into a folder where gnu-linux user postgres is owner su - root mkdir /backup cp -rv 2020-08-10-09.backup.postgresql /backup chown -R postgres: /backup su - postgres -c "psql -f /backup/2020-08-10-09.backup.postgresql;" # backup/export/dump database "dbname" # will create filename like: pg_dump -U dbuser -h localhost dbname > /backup/postgres/$(date '+%Y-%m-%d_%H_%M_%S')_dbname.sql # how to import backup/database dump psql -U dbuser dbname < /backup/postgres/2020-05-25_23_18_11_dbname.sql # or if the dump is in dump.sql.gz packed format gunzip -c /path/to/dump.sql.gz | psql -U postgres # more documentation on export/import database backups https://www.postgresql.org/docs/9.1/backup-dump.html
multi threaded backup and restore
$DB_TO_BACKUP=DatabaseName # backup single database multi threaded (10x threads) time su postgres -c "pg_dump -j 10 -Fd -Z 9 --file=$(date '+%Y-%m-%d').$DB_TO_BACKUP.postgres_dump_directory_format.compressed $DB_TO_BACKUP"; # restore single database multi threaded (10x threads) $DB_TO_BACKUP=DatabaseName createdb -U postgres $DB_TO_BACKUP time su postgres -c "pg_restore -Fd -j 10 -d $DB_TO_BACKUP -U postgres $(date '+%Y-%m-%d').$DB_TO_BACKUP.postgres_dump_directory_format.compressed"
enable detailed logs
#!WARNING! THIS WAS TESTED FOR OLDER RHEL7 AND CENTOS!
# manual editing vim /var/lib/pgsql/12/data/postgresql.conf; # postgres main config file # semi-automatic editing via sed # optional: postgres: enable detailed logging POSTGRES_MAIN_CONFIG="/var/lib/pgsql/12/data/postgresql.conf" sed -i 's/#log_statement = [^"]*/log_statement = 'all'/g' $POSTGRES_MAIN_CONFIG sed -i 's/#logging_collector = [^"]*/logging_collector = on/g' $POSTGRES_MAIN_CONFIG sed -i 's/#log_connections = [^"]*/log_connections = on/g' $POSTGRES_MAIN_CONFIG sed -i 's/#log_disconnections = [^"]*/log_disconnections = on/g' $POSTGRES_MAIN_CONFIG sed -i 's/#log_error_verbosity = [^"]*/log_error_verbosity = default/g' $POSTGRES_MAIN_CONFIG # even more verbose logging of details # this is VERY VERY detailed sed -i 's/#log_error_verbosity = [^"]*/log_error_verbosity = verbose/g' $POSTGRES_MAIN_CONFIG systemctl restart postgresql-12.service; # restart to make config changes active # logs should now output here ==> /var/lib/pgsql/12/data/log/postgresql-Mon.log <== ==> /var/lib/pgsql/12/data/log/postgresql-Tue.log <== # let's monitor those vim /scripts/monitor.sh #!/bin/bash # version: 1.1 # description: this script will allow to see changes on the system in real time, it allows monitoring multiple log files at the same time # usage: /path/to/script/monitor.sh # monitoring all files below /var/log find /var/log/ -type f \( -name "*" \) ! -path *.gz* -exec tail -n0 -f "$file" {} + & # monitoring postgres logs (please enable logging in /var/lib/pgsql/12/data/postgresql.conf manually find /var/lib/pgsql/12/data/log/ -type f \( -name "*" \) ! -path *.gz* -exec tail -n0 -f "$file" {} + & # monitoring all files.log below /var/opt find /var/opt/ -type f \( -name "*log*" \) ! -path *.gz* -exec tail -n0 -f "$file" {} +
script: display information about the installed postgres instance: show info about postgres database, files, config and usage stats
- … postgres: installed version?
su - rootecho "... postgres: installed version?" su - postgres -c "psql -d postgres -c \"SELECT version();\"" echo "... postgres: where is the main config file?" su - postgres -c "psql -d postgres -c \"SHOW config_file;\"" echo "... postgres: where is the main data directory? (where files are stored)" su - postgres -c "psql -d postgres -c \"SHOW data_directory;\"" echo "... postgres: usage stats?" su - postgres -c " psql -d postgres -c \" SELECT 'Database Name' AS database, pg_database.datname AS database, pg_size_pretty(pg_database_size(pg_database.datname)) AS size, pg_stat_database.numbackends AS connections, pg_stat_database.xact_commit AS commits, pg_stat_database.xact_rollback AS rollbacks, pg_stat_database.blks_read AS blocks_read, pg_stat_database.blks_hit AS blocks_hit, pg_stat_database.tup_returned AS rows_returned, pg_stat_database.tup_fetched AS rows_fetched, pg_stat_database.tup_inserted AS rows_inserted, pg_stat_database.tup_updated AS rows_updated, pg_stat_database.tup_deleted AS rows_deleted FROM pg_database JOIN pg_stat_database ON pg_database.datname = pg_stat_database.datname WHERE pg_database.datistemplate = false ORDER BY pg_database.datname;\" "
alternativels download the script: postgres_info.sh.txt
move postgres db to differen harddisk or partition or filesystem
systemctl stop postgresql rsync -r -vv --update --progress /var/lib/postgresql/12/main/ /media/user/data/postgres sed -i.bak "s|data_directory = '/var/lib/postgresql/12/main'|data_directory = '/media/user/data/postgres'|" /etc/postgresql/12/main/postgresql.conf # give user postgres access chown -R postgres:postgres /media/user/ chmod 777 /media/user/ chmod 777 /media/user/data/ # or postgres will complain chmod 0750 /media/user/data/postgres # make sure to enable detailed logging (see above) # and watch all logs closely during startup process as errors might occur systemctl start postgresql
benchmarks?
- not sure how “legit” it is, this benchmark will create create a db test_benchmark with 10x tables and create 10x dump.sql and import that one by one…
- postgres_bench_v2.sh.txt
- result: inside a kvm vm this bench took 2m56.246s which means
- echo “scale=5; 10000*10/(176)”|bc
568 inserts per sec (IOPS?) (with no significant usage of CPU, RAM or DISC this means: the bottleneck is somewhere else)
- postgres_bench_v2.sh.txt
liked this article?
- only together we can create a truly free world
- plz support dwaves to keep it up & running!
- (yes the info on the internet is (mostly) free but beer is still not free (still have to work on that))
- really really hate advertisement
- contribute: whenever a solution was found, blog about it for others to find!
- talk about, recommend & link to this blog and articles
- thanks to all who contribute!
