Log analysis adalah skill fundamental untuk system administrator. Dengan command line tools yang tepat, Anda dapat mengextract insights dari log files, mendeteksi anomalies, dan troubleshoot masalah dengan efisien.

1. Tools untuk Log Analysis

Essential Log Commands

# View logs in real-time
tail -f /var/log/syslog
tail -f /var/log/nginx/access.log
View recent logs
tail -n 100 /var/log/auth.log
View from specific line
tail -n +1000 /var/log/syslog | head -n 100
View compressed logs
zcat /var/log/syslog.1.gz | tail -n 100
zless /var/log/syslog.1.gz
Search dalam compressed logs
zgrep "error" /var/log/syslog.*.gz

Journalctl (untuk systemd)

# View all logs
journalctl
View logs in real-time
journalctl -f
View specific service logs
journalctl -u nginx
journalctl -u ssh
View today logs
journalctl --since today
View last hour
journalctl --since "1 hour ago"
View specific time range
journalctl --since "2026-02-01 10:00:00" --until "2026-02-01 12:00:00"
View specific priority
journalctl -p err
journalctl -p warning
View kernel logs
journalctl -k
View boot logs
journalctl -b
Output formats
journalctl -o json
journalctl -o short-iso

2. Pattern Matching dengan Grep

Basic Grep Operations

# Search for errors
grep "error" /var/log/syslog
Case insensitive search
grep -i "error" /var/log/syslog
Show line numbers
grep -n "error" /var/log/syslog
Count occurrences
grep -c "error" /var/log/syslog
Show context lines
grep -C 3 "error" /var/log/syslog  # 3 lines before and after
grep -B 5 "error" /var/log/syslog   # 5 lines before
grep -A 5 "error" /var/log/syslog   # 5 lines after
Invert match (show lines yang TIDAK match)
grep -v "error" /var/log/syslog
Multiple patterns
grep -E "error|warning|critical" /var/log/syslog
grep -e "error" -e "failed" /var/log/syslog
Regex patterns
grep -E "[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}" /var/log/nginx/access.log

Advanced Grep untuk Log Analysis

# Find failed SSH login attempts
grep "Failed password" /var/log/auth.log
Find dengan specific username
grep "Failed password for admin" /var/log/auth.log
Extract IP addresses
grep -oE "[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}" /var/log/auth.log
Find specific HTTP status codes
grep '" 404 ' /var/log/nginx/access.log
grep '" 500 ' /var/log/nginx/access.log
Search dalam multiple files
grep "error" /var/log/*.log
grep -r "error" /var/log/
Find files yang mengandung pattern
grep -l "error" /var/log/*.log
Search dengan exclude files
grep "error" /var/log/
.log --exclude=".gz"

3. Log Analysis dengan AWK

Parsing Structured Logs

# Apache/Nginx access log analysis
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20
Count requests per IP
awk '{ip[$1]++} END {for(i in ip) print ip[i], i}' /var/log/nginx/access.log | sort -rn | head -20
Calculate total bandwidth per IP
awk '{bytes[$1] += $10} END {for(i in bytes) print bytes[i], i}' /var/log/nginx/access.log | sort -rn | head -20
HTTP status code analysis
awk '{code[$9]++} END {for(c in code) print code[c], c}' /var/log/nginx/access.log | sort -rn
User agent analysis
awk -F'"' '{print $6}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -10
Request URL analysis
awk '{url[$7]++} END {for(u in url) print url[u], u}' /var/log/nginx/access.log | sort -rn | head -20
Response time analysis
awk '{sum += $11; count++} END {print "Average:", sum/count "ms"}' /var/log/nginx/access.log

System Log Analysis

# Error frequency per hour
awk '/error/ {hour=substr($3,1,2); count[hour]++} END {for(h in count) print h ":00", count[h]}' /var/log/syslog
Memory usage trend
awk '/MemAvailable/ {gsub(/kB/,""); print $2}' /var/log/syslog
Disk space alerts
grep "No space left" /var/log/syslog
Service restart analysis
grep -E "Starting|Stopping|Restarting" /var/log/syslog | awk '{print $5, $6, $7}' | sort | uniq -c | sort -rn

4. Processing dengan Sed

Text Transformation

# Extract specific fields
sed -n 's/.*\[client \([^]]*\)\].*/\1/p' /var/log/apache2/error.log
Remove timestamp untuk easier pattern matching
sed 's/^[^ ]
 [^ ] //' /var/log/syslog | grep "error"
Format output
sed -n 's/.
[error] (.)/ERROR: \1/p' /var/log/apache2/error.log
Filter dan transform
sed '/error/!d; s/.*error: //' /var/log/syslog
Remove duplicate consecutive lines
sed '$!N; /^(.*)\n\1$/!P; D' /var/log/syslog

5. Advanced Log Analysis

Log Rotation dan Management

# Check log sizes
du -sh /var/log/*.log | sort -h
Find large logs
find /var/log -name "*.log" -size +100M
Compress old logs
find /var/log -name "*.log" -mtime +7 -exec gzip {} \;
Clean old compressed logs
find /var/log -name "*.gz" -mtime +30 -delete
Check logrotate status
cat /var/lib/logrotate/status

Security Log Analysis

# Failed login attempts
lastb | head -20
grep "Failed password" /var/log/auth.log | wc -l
Successful logins
last | head -20
grep "Accepted password" /var/log/auth.log
SSH brute force attempts
awk '/Failed password/ {print $11}' /var/log/auth.log | sort | uniq -c | sort -rn | head -10
Find suspicious activity
grep -E "(Invalid user|Failed password|Connection closed|Received disconnect)" /var/log/auth.log | tail -50
Sudo usage analysis
grep "sudo:" /var/log/auth.log | grep -v "COMMAND=/usr/bin/sudo" | tail -20
Check for root login attempts
grep "root" /var/log/auth.log | grep "Failed"

Application Log Analysis

# MySQL slow queries
tail -n 50 /var/log/mysql/slow.log
PHP errors
grep -i "error|warning|fatal" /var/log/php_errors.log
Application specific
grep -E "(Exception|Error|Fatal|Crash)" /var/www/app/logs/app.log
Database connection errors
grep "Connection refused|Too many connections" /var/log/mysql/error.log
Queue worker logs
grep "Processing|Processed|Failed" /var/log/worker.log

6. Real-time Monitoring dan Alerting

Real-time Log Monitoring

# Monitor multiple logs
tail -f /var/log/syslog /var/log/nginx/access.log
Monitor dengan filter
tail -f /var/log/syslog | grep -E "error|warning|critical"
Monitor dengan highlight
tail -f /var/log/syslog | grep --color=always -E "error|$"
Log monitoring dengan timestamps
tail -f /var/log/syslog | while read line; do echo "[$(date '+%H:%M:%S')] $line"; done
Multi-file monitoring dengan labels
tail -f /var/log/nginx/access.log | sed 's/^/[NGINX] /' &
tail -f /var/log/php_errors.log | sed 's/^/[PHP] /' &
wait

Simple Log Alerting

#!/bin/bash
# log-alert.sh
LOG_FILE="/var/log/syslog"
ALERT_PATTERN="critical|emergency|kernel panic"
EMAIL="[email protected]"
Monitor dan alert
tail -n 0 -f $LOG_FILE | while read line; do
if echo "$line" | grep -iE "$ALERT_PATTERN"; then
echo "ALERT: $line" | mail -s "Server Alert" $EMAIL
fi
done

Kesimpulan

Log analysis adalah skill essential untuk setiap system administrator. Dengan menguasai grep, awk, sed, dan journalctl, Anda dapat mengextract insights dari log files dengan efisien.

Best Practices:
1. Archive logs older than 30 days
2. Monitor logs secara real-time untuk critical services
3. Set up automated alerts untuk error patterns
4. Use structured logging jika memungkinkan
5. Implement centralized logging untuk multiple servers
6. Regular log rotation untuk manage disk space

Tools Alternatif:
– GoAccess: Real-time web log analyzer
– Logwatch: Automated log analysis reporting
– Graylog/OpenSearch: Centralized logging
– ELK Stack: Enterprise log analysis

Command Reference:
– tail -f: Real-time monitoring
– grep -E: Extended regex search
– awk: Field extraction dan calculations
– sed: Text transformation
– journalctl: Systemd log analysis