EKS Cloudwatch - Container Insight + Fluentd 설치
아래 포스팅과 Container Insight 및 Fluentd 부분은 url 부분과 설정 yaml이 상이함
slack 및 Whatap 및 slack 연동 + 알림 설정 하고 싶다면 하기 게시물 참고
https://hiheey.tistory.com/171
CloudWatch Container Insight 설정 후, Whatap 및 slack 연동 + 알림 설정
hiheey.tistory.com
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-metrics.html
밑에 권한을 추가하라고 해서 하기는 했는데 노드 그룹에 권한 추가만 해주면 될 것 같다
필수라고 기재했으니 참고!
<이 부분은 선택적 권한>
생성된 역할에 권한 추가
JSON으로 바꾸고 하기와 같이 권한 변경
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "logs:PutRetentionPolicy",
"Resource": "*"
}
]
}
iam > 사용자 부분
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "logs:PutRetentionPolicy",
"Resource": "*"
}
]
}
필수!!
Node 그룹 역할에 CloudWatchAgentServerPolicy 권한 추가!
이거 안해주면 무수한 access denied를 영접하게 된다...
collectd 설치 (CloudWatch 에이전트의 구성 파일이 collectd 사용)
apt-get update
apt-get install collectd
<옵션>
amazon-cloudwatch-agent가 실행된 상태라면 수정 사항 적용
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a stop
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a start
<방법 1>
워커 노드에 iam 권한 부여
aws iam attach-role-policy \ --role-name 역할 이름 \ --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
나의 경우 worker node 의 역할을 기입
<방법 2>
서비스 계정 사용
eksctl create iamserviceaccount \
--name cloudwatch-agent \
--namespace amazon-cloudwatch --cluster <클러스터명> \
--role-name <생성할 역할이름> \
--attach-policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
--role-only \
--approve
wget https://amazoncloudwatch-agent.s3.amazonaws.com/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb
dpkg -i -E ./amazon-cloudwatch-agent.deb
json 파일 생성
원하는 사항에 체크하여 생성
나의 경우 path 는 /var/log로 지정
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
파일 생성 확인
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:config.json
네임스페이스 생성
서비스어카운트 생성
configmap 다운로드 및 설정
cwagent-configmap.yaml
# create configmap for cwagent config
apiVersion: v1
data:
# Configuration is in Json format. No matter what configure change you make,
# please keep the Json blob valid.
cwagentconfig.json: |
{
"logs": {
"metrics_collected": {
"kubernetes": {
"cluster_name": "kang-cluster", # 이 부분은 클러스터명
"metrics_collection_interval": 60
}
},
"force_flush_interval": 5
}
}
kind: ConfigMap
metadata:
name: cwagentconfig
namespace: amazon-cloudwatch
curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-daemonset.yaml
환경변수 지정
ClusterName=kang-cluster
RegionName=ap-northeast-2
FluentBitHttpPort='2020'
FluentBitReadFromHead='Off'
[[ ${FluentBitReadFromHead} = 'On' ]] && FluentBitReadFromTail='Off'|| FluentBitReadFromTail='On'
[[ -z ${FluentBitHttpPort} ]] && FluentBitHttpServer='Off' || FluentBitHttpServer='On'
kubectl create configmap fluent-bit-cluster-info \
--from-literal=cluster.name=${ClusterName} \
--from-literal=http.server=${FluentBitHttpServer} \
--from-literal=http.port=${FluentBitHttpPort} \
--from-literal=read.head=${FluentBitReadFromHead} \
--from-literal=read.tail=${FluentBitReadFromTail} \
--from-literal=logs.region=${RegionName} -n amazon-cloudwatch
cwagent-daemonset.yaml
19~22번째 줄 주석 처리
# deploy cwagent as daemonset
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: cloudwatch-agent
namespace: amazon-cloudwatch
spec:
selector:
matchLabels:
name: cloudwatch-agent
template:
metadata:
labels:
name: cloudwatch-agent
spec:
containers:
- name: cloudwatch-agent
image: public.ecr.aws/cloudwatch-agent/cloudwatch-agent:1.300030.2b309
ports:
- containerPort: 8125
hostPort: 8125
protocol: UDP
resources:
limits:
cpu: 400m
memory: 400Mi
requests:
cpu: 400m
memory: 400Mi
# Please don't change below envs
env:
- name: HOST_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
- name: HOST_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: K8S_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: CI_VERSION
value: "k8s/1.3.17"
# Please don't change the mountPath
volumeMounts:
- name: cwagentconfig
mountPath: /etc/cwagentconfig
- name: rootfs
mountPath: /rootfs
readOnly: true
- name: dockersock
mountPath: /var/run/docker.sock
readOnly: true
- name: varlibdocker
mountPath: /var/lib/docker
readOnly: true
- name: containerdsock
mountPath: /run/containerd/containerd.sock
readOnly: true
- name: sys
mountPath: /sys
readOnly: true
- name: devdisk
mountPath: /dev/disk
readOnly: true
nodeSelector:
kubernetes.io/os: linux
volumes:
- name: cwagentconfig
configMap:
name: cwagentconfig
- name: rootfs
hostPath:
path: /
- name: dockersock
hostPath:
path: /var/run/docker.sock
- name: varlibdocker
hostPath:
path: /var/lib/docker
- name: containerdsock
hostPath:
path: /run/containerd/containerd.sock
- name: sys
hostPath:
path: /sys
- name: devdisk
hostPath:
path: /dev/disk/
terminationGracePeriodSeconds: 60
serviceAccountName: cloudwatch-agent
생성 확인
