#Software

縮寫

  • SDLC(Software Development Life Cycle,軟體開發生命週期)

名詞用法

State Status
定義 表示具備明確轉換流程的狀態 表示單一、靜態的狀態或屬性
轉換關係 狀態之間有明確的遷移關係 狀態之間沒有明確的轉換流程
範例 TCP 狀態(SYN_SENT、ESTABLISHED 等) HTTP 狀態碼(200、404 等)、啟用/停用標記

Project Management

Meaning of Abbreviations

Shortcut Description
PR Pull Request
PTAL Please Take A Look
TBR To Be Reviewed
TL;DR Too Long; Didn’t Read
LGTM Looks Good To Me
SGTM Sounds Good To Me
TBD To Be Done

Environment

#Python

Design Pattern

基本
State Design Pattern
  1. The State Design Pattern in Python Explained
Class
  1. Python dataclass 教學:輕鬆定義資料類別
  2. Python Property 教學:保護變數資料的 Getter 與 Setter
  3. Python Type Hints 教學:我犯過的 3 個菜鳥錯誤
  4. Pydantic: Simplifying Data Validation in Python
Enum
  1. Python Enum 枚舉教學:有條理管理常數集合的好工具
Documentation
  1. mkdocstrings/griffe
  2. Mintlify

    Meet the next generation of documentation. AI-native, beautiful out-of-the-box, and built for developers

  3. Top 6 Tools to Turn Code into Beautiful Diagrams
  4. Diagrams as Code
Project
  1. astral-sh/uv

    An extremely fast Python package installer and resolver, written in Rust.

  2. astral-sh/rye

    a Hassle-Free Python Experience

  3. cookiecutter

    A cross-platform command-line utility that creates projects from cookiecutters (project templates), e.g. Python package projects, C projects.

  4. pypa/hatch

    Modern, extensible Python project management

Flow Chart
  1. cdfmlr/pyflowchart
  2. PYTHON 自動產生 FLOWCHART !
Pipeline
  1. Data PIPELINE - Orchest
  2. airflow使用指南-机器学习工程自动化
Docker

減少 Image 大小

參數 解釋
PIP_DISABLE_PIP_VERSION_CHECK=1 關閉 pip 在執行時檢查「pip 本身是否有新版」的提示
PIP_NO_CACHE_DIR=1 讓 pip 安裝套件時不要保留下載的 wheel 或壓縮檔快取
PYTHONDONTWRITEBYTECODE=1 讓 Python 不要在執行時生成 .pyc / __pycache__ 檔案
PYTHONUNBUFFERED=1 讓 Python 輸出(stdout/stderr)不經過快取,直接 flush

#Julia

#SRE Site Reliability Engineering

Termindology

  1. Chaos Engineering - 混沌工程

#CI/CD

Termindology

  1. Infrastructure as Code IaC
  2. GitOps

Version Control

Distributed Version Control System,DVCS
Center Version Control System
Team Foundation Version Control, TFVC
git
DVC

ML Experiments Management with Git

  1. Data Version Control | Git for Data & Models | ML Experiments Management
  2. Data Version Control With Python and DVC
elementary

Open-source data observability for analytics engineers

  1. https://github.com/elementary-data/elementaryelementary

Workflow/Pipeline/MLops

Crontab
Jenkins
Github action
Airflow
mage-ai

The modern replacement for Airflow. Build, run, and manage data pipelines for integrating and transforming data.

  1. https://github.com/mage-ai/mage-ai
wandb

A tool for visualizing and tracking your machine learning experiments.

  1. https://github.com/wandb/wandb
Drone CI
Flyte
  1. An open-source orchestrator that facilitates building production-grade data and ML pipelines
datachecks

Datachecks is an open-source data monitoring tool that helps to monitor the data quality of databases and data pipelines. It identifies potential issues, including in the databases and data pipelines. It helps to identify the root cause of the data quality issues and helps to improve the data quality.

  1. https://github.com/waterdipai/datachecks

Logs

hydra

A framework for elegantly configuring complex applications

  1. https://github.com/facebookresearch/hydra
whylogs

The open standard for data logging

  1. https://github.com/whylabs/whylogs

#git

Command

# 列出所有曾經有 commit 過的文件
git log --pretty=format: --name-only --diff-filter=A | sort -u
# 刪除所有跟 a.txt 相關的紀錄
git filter-branch --force --index-filter 'git rm --cached --ignore-unmatch a.txt' --prune-empty --tag-name-filter cat -- --all

Commit message Convention

Type Description
feat Add new features
chore Build process or auxiliary tool changes
fix Fix some bugs
refactor A code change that neither fixes a bug or adds a feature
style Change coding style
docs Write new documentation

Branch Naming Convention

Type Description
wip Works in progress
feat Feature
bug Bug fix or experiment
junk Throwaway branch created to experiment

Rebase

Commands Description
pick 保留該提交,不進行任何更改
reword 修改該提交的提交訊息
edit 挑選該提交,並在應用該提交時停止,以便你可以進行修改
squash 將該提交與前一個提交合併成一個提交
fixup 與 squash 類似,但用於合併提交而無需保留該提交的提交訊息
exec 執行一個 shell 命令。這允許你在 rebase 過程中執行自定義的命令