[{"data":1,"prerenderedAt":711},["ShallowReactive",2],{"/en-us/blog/tracking-down-missing-tcp-keepalives/":3,"navigation-en-us":42,"banner-en-us":458,"footer-en-us":473,"Stan Hu":683,"next-steps-en-us":696},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"seo":8,"content":16,"config":32,"_id":35,"_type":36,"title":37,"_source":38,"_file":39,"_stem":40,"_extension":41},"/en-us/blog/tracking-down-missing-tcp-keepalives","blog",false,"",{"title":9,"description":10,"ogTitle":9,"ogDescription":10,"noIndex":6,"ogImage":11,"ogUrl":12,"ogSiteName":13,"ogType":14,"canonicalUrls":12,"schema":15},"Tracking TCP Keepalives: Lessons in Docker, Golang & GitLab","An in-depth recap of debugging a bug in the Docker client library.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1749680874/Blog/Hero%20Images/network.jpg","https://about.gitlab.com/blog/tracking-down-missing-tcp-keepalives","https://about.gitlab.com","article","\n                        {\n        \"@context\": \"https://schema.org\",\n        \"@type\": \"Article\",\n        \"headline\": \"What tracking down missing TCP Keepalives taught me about Docker, Golang, and GitLab\",\n        \"author\": [{\"@type\":\"Person\",\"name\":\"Stan Hu\"}],\n        \"datePublished\": \"2019-11-15\",\n      }",{"title":17,"description":10,"authors":18,"heroImage":11,"date":20,"body":21,"category":22,"tags":23},"What tracking down missing TCP Keepalives taught me about Docker, Golang, and GitLab",[19],"Stan Hu","2019-11-15","\n\nThis blog post was originally published on the GitLab Unfiltered blog. It was reviewed and republished on 2019-12-03.\n{: .alert .alert-info .note}\n\nWhat began as failure in a GitLab static analysis check led to a\ndizzying investigation that uncovered a subtle [bug in the Docker client\nlibrary code](https://github.com/docker/for-linux/issues/853) used by\nthe GitLab Runner. We ultimately worked around the problem by upgrading\nthe Go compiler, but in the process we uncovered an unexpected change in\nthe Go TCP keepalive defaults that fixed an issue with Docker and GitLab\nCI.\n\nThis investigation started on October 23, when backend engineer [Luke\nDuncalfe](/company/team/#.luke) mentioned, \"I'm seeing\n[`static-analysis` failures with no output](https://gitlab.com/gitlab-org/gitlab/-/jobs/331174397).\nIs there something wrong with this job?\" He opened [a GitLab\nissue](https://gitlab.com/gitlab-org/gitlab/issues/34951) to discuss.\n\nWhen Luke ran the static analysis check locally on his laptop, he saw\nuseful debugging output when the test failed. For example, an extraneous\nnewline would accurately be reported by Rubocop. However, when the same\ntest ran in GitLab's automated test infrastructure, the test failed\nquietly:\n\n![Failed job](https://about.gitlab.com/images/blogimages/docker-tcp-keepalive-debug/job-failure.png){: .shadow.center}\n\nNotice how the job log did not include any clues after the `bin/rake\nlint:all` step. This made it difficult to determine whether a real\nproblem existed, or whether this was just a flaky test.\n\nIn the ensuing days, numerous team members reported the same problem.\nNothing kills productivity like silent test failures.\n\n## Was something wrong with the test itself?\n\nIn the past, we had seen that if that specific test generated enough\nerrors, [the output buffer would fill up, and the continuous integration\n(CI) job would lock\nindefinitely](https://gitlab.com/gitlab-org/gitlab-foss/issues/61432). We\nthought we had [fixed that issue months\nago](https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/28402). Upon\nfurther review, that fix seemed to eliminate any chance of a thread\ndeadlock.\n\nDid we have to flush the buffer? No, because the Linux kernel will do\nthat for an exiting process already.\n\n## Was there a change in how CI logs were handled?\n\nWhen a test runs in GitLab CI, the [GitLab\nRunner](https://gitlab.com/gitlab-org/gitlab-runner/) launches a Docker\ncontainer that runs commands specified by a `.gitlab-ci.yml` inside the\nproject repository. As the job runs, the runner streams the output to\nthe GitLab API via PATCH requests. The GitLab backend saves this data\ninto a file. The following sequence diagram shows how this works:\n\n```plantuml\n== Get a job! ==\nRunner -> GitLab: POST /api/v4/jobs/request\nGitLab -> Runner: 201 Job was scheduled\n\n== Job sends logs (1 of 2) ==\nRunner -> GitLab: PATCH /api/v4/job/:id/trace\nGitLab -> File: Save to disk\nGitLab -> Runner: 202 Accepted\n\n== Job sends logs (2 of 2) ==\nRunner -> GitLab: PATCH /api/v4/job/:id/trace\nGitLab -> File: Save to disk\nGitLab -> Runner: 202 Accepted\n```\n\n[Henrich Lee Yu](/company/team/#engwan) mentioned\nthat we had recently [disabled a feature flag that changed how GitLab\nhandled CI job\nlogs](https://docs.gitlab.com/ee/administration/job_logs.html#new-incremental-logging-architecture). [The\ntiming seemed to line\nup](https://gitlab.com/gitlab-org/gitlab/issues/34951#note_236723888).\n\nThis feature, called live CI traces, eliminates the need for a shared\nPOSIX filesystem (e.g., NFS) when saving job logs to disk by:\n\n1. Streaming data into memory via Redis\n2. Persisting the data in the database (PostgreSQL)\n3. Archiving the final data into object storage\n\nWhen this flag is enabled, the flow of CI job logs looks something like\nthe following:\n\n```plantuml\n== Get a job! ==\nRunner -> GitLab: POST /api/v4/jobs/request\nGitLab -> Runner: 201 Job was scheduled\n\n== Job sends logs ==\nRunner -> GitLab: PATCH /api/v4/job/:id/trace\nGitLab -> Redis: Save chunk\nGitLab -> Runner: 202 Accepted\n...\n== Copy 128 KB chunks from Redis to database ==\nGitLab -> Redis: GET gitlab:ci:trace:id:chunks:0\nGitLab -> PostgreSQL: INSERT INTO ci_build_trace_chunks\n...\n== Job finishes ==\n\nRunner -> GitLab: PUT /api/v4/job/:id\nGitLab -> Runner: 200 Job was updated\n\n== Archive trace to object storage ==\n```\n\nLooking at the flow diagram above, we see that this approach has more\nsteps. After receiving data from the runner, something could have gone\nwrong with handling a chunk of data. However, we still had many\nquestions:\n\n1. Did the runners send the right data in the first place?\n1. Did GitLab drop a chunk of data somewhere?\n1. Did this new feature actually have anything to do with the problem?\n1. Are they really making another Gremlins movie?\n\n## Reproducing the bug: Simplify the `.gitlab-ci.yml`\n\nTo help answer those questions, we simplified the `.gitlab-ci.yml` to\nrun only the `static-analysis` step. We inserted a known Rubocop error,\nreplacing a `eq` with `eql`. We first ran this test on a separate GitLab\ninstance with a private runner. No luck there – the job showed the right\noutput:\n\n```\nOffenses:\n\nee/spec/models/project_spec.rb:55:42: C: RSpec/BeEql: Prefer be over eql.\n        expect(described_class.count).to eql(2)\n                                         ^^^\n\n12669 files inspected, 1 offense detected\n```\n\nHowever, we repeated the test on our staging server and found that we\nreproduced the original problem. In addition, the live CI trace feature\nflag had been activated on staging. Since the problem occurred with and\nwithout the feature, we could eliminate that feature as a possible\ncause.\n\nPerhaps something with the GitLab server environment caused a\nproblem. For example, could the load balancers be rate-limiting the\nrunners? As an experiment, we pointed a private runner at the staging\nserver and re-ran the test. This time, it succeeded: the output was\nshown. That seemed to suggest that the problem had more to do with the\nrunner than with the server.\n\n## Docker Machine vs. Docker\n\nOne key difference between the two tests: One runner used a shared,\nautoscaled runner using a [Docker\nMachine](https://docs.docker.com/machine/overview/) executor, and the\nprivate runner used a [Docker\nexecutor](https://docs.gitlab.com/runner/executors/docker.html).\n\nWhat does Docker Machine do exactly? The following diagram may help\nillustrate:\n\n![Docker Machine](https://docs.docker.com/machine/img/machine.png){: .medium.center}\n\nThe top-left shows a local Docker instance. When you run Docker from the\ncommand-line interface (e.g., `docker attach my-container`), the program\njust makes [REST calls to the Docker Engine\nAPI](https://docs.docker.com/engine/api/v1.40/).\n\nThe rest of the diagram shows how Docker Machine fits into the\npicture. Docker Machine is an entirely separate program. The GitLab\nRunner shells out to `docker-machine` to create and destroy virtual\nmachines using cloud-specific (e.g. Amazon, Google, etc.) drivers. Once\na machine is running, the runner then uses the Docker Engine API to run,\nwatch, and stop containers.\n\nNote that this API is used securely over an HTTPS connection. This is an\nimportant difference between the Docker Machine executor and Docker\nexecutor: The former needs to communicate across the network, while the\nlatter can either use a local TCP socket or UNIX domain socket.\n\n## Google Cloud Platform timeouts\n\nWe've known for a while that Google Cloud [has a 10-minute idle\ntimeout](https://cloud.google.com/compute/docs/troubleshooting/general-tips),\nwhich has caused issues in the past:\n\n> Note that idle connections are tracked for a maximum of 10 minutes,\n> after which their traffic is subject to firewall rules, including the\n> implied deny ingress rule. If your instance initiates or accepts\n> long-lived connections with an external host, you should adjust TCP\n> keep-alive settings on your Compute Engine instances to less than 600\n> seconds to ensure that connections are refreshed before the timeout\n> occurs.\n\nWas the problem caused by this timeout? With the Docker Machine\nexecutor, we found that we could reproduce the problem with a simple\n`.gitlab-ci.yml`:\n\n```yaml\nimage: \"busybox:latest\"\n\ntest:\n  script:\n    - date\n    - sleep 601\n    - echo \"Hello world!\"\n    - date\n    - exit 1\n```\n\nThis would reproduce the failure, where we would never see the `Hello\nworld!` output. Changing the `sleep 601` to `sleep 599` would make the\nproblem go away. Hurrah! All we have to do is tweak the system TCP\nkeepalives, right? Google provided these sensible settings:\n\n```sh\nsudo /sbin/sysctl -w net.ipv4.tcp_keepalive_time=60 net.ipv4.tcp_keepalive_intvl=60 net.ipv4.tcp_keepalive_probes=5\n```\n\nHowever, enabling these kernel-level settings didn't solve the\nproblem. Were keepalives even being sent? Or was there some other issue?\nWe turned our attention to network traces.\n\n## Eavesdropping on Docker traffic\n\nIn order to understand what was happening, we needed to be able to\nmonitor the network communication between the runner and the Docker\ncontainer. But how exactly does the GitLab Runner stream data from a\nDocker container to the GitLab server?  The following diagram\nillustrates the flow:\n\n```plantuml\nRunner -> Docker: POST /containers/name/attach\nDocker -> Runner: \u003Ccontainer output>\nDocker -> Runner: \u003Ccontainer output>\nRunner -> GitLab: PATCH /api/v4/job/:id/trace\nGitLab -> File: Save to disk\nGitLab -> Runner: 202 Accepted\n```\n\nFirst, the runner makes a [POST request to attach to the container\noutput](https://docs.docker.com/engine/api/v1.40/#operation/ContainerAttach).\nAs soon as a process running in the container outputs some data, Docker\nwill transmit the data over this HTTPS stream. The runner then copies\nthis data to GitLab via the PATCH request.\n\nHowever, as mentioned earlier, traffic between a GitLab Runner and the\nremote Docker machine is encrypted over HTTPS on port 2376. Was there an\neasy way to disable HTTPS? Searching through the code of Docker Machine,\nwe found that it did not appear to be supported out of the box.\n\nSince we couldn't disable HTTPS, we had two ways to eavesdrop:\n\n1. Use a man-in-the-middle proxy (e.g. [mitmproxy](https://mitmproxy.org/))\n1. Record the traffic and decrypt the traffic later using the private keys\n\n## Ok, let's be the man-in-the-middle!\n\nThe first seemed more straightforward, since [we already had experience\ndoing this with the Docker\nclient](https://docs.gitlab.com/ee/administration/packages/container_registry.html#running-the-docker-daemon-with-a-proxy).\n\nHowever, after [defining the proxy variables for GitLab\nRunner](https://docs.gitlab.com/runner/configuration/proxy.html#adding-proxy-variables-to-the-runner-config),\nwe found we were only able to intercept the GitLab API calls with\n`mitmproxy`. The Docker API calls still went directly to the remote\nhost. Something wasn't obeying the proxy configuration, but we didn't\ninvestigate further. We tried the second approach.\n\n## Decrypting TLS data\n\nTo decrypt TLS data, we would need to obtain the encryption keys. Where\nwere these located for a newly-created system with `docker-machine`? It\nturns out `docker-machine` worked in the following way:\n\n1. Call the Google Cloud API to create a new machine\n1. Create a `/root/.docker/machine/machines/:machine_name` directory\n1. Generate a new SSH keypair\n1. Install the SSH key on the server\n1. Generate a new TLS certificate and key\n1. Install and configure Docker on the newly-created machine with TLS certificates\n\nAs long as the machine runs, the directory will contain the information\nneeded to decode this traffic. We ran `tcpdump` and saved the private keys.\n\nOur first attempt at decoding the traffic failed. Wireshark could not\ndecode the encrypted traffic, although general TCP traffic could still\nbe seen. Researching more, we found out why: If the encrypted traffic\nused a [Diffie-Hellman key\nexchange](https://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exchange),\nhaving the private keys would not suffice! This is by design, a property\ncalled [perfect forward\nsecrecy](https://en.m.wikipedia.org/wiki/Forward_secrecy).\n\nTo get around that limitation, we modified the GitLab Runner to disable\ncipher suites that used the Diffie-Hellman key exchange:\n\n```diff\ndiff --git a/vendor/github.com/docker/go-connections/tlsconfig/config_client_ciphers.go b/vendor/github.com/docker/go-connections/tlsconfig/config_client_ciphers.go\nindex 6b4c6a7c0..a3f86d756 100644\n","engineering",[24,25,26,27,28,27,29,30,31],"community","git","GitOps","CI","google","AWS","testing","features",{"slug":33,"featured":6,"template":34},"tracking-down-missing-tcp-keepalives","BlogPost","content:en-us:blog:tracking-down-missing-tcp-keepalives.yml","yaml","Tracking Down Missing Tcp Keepalives","content","en-us/blog/tracking-down-missing-tcp-keepalives.yml","en-us/blog/tracking-down-missing-tcp-keepalives","yml",{"_path":43,"_dir":44,"_draft":6,"_partial":6,"_locale":7,"data":45,"_id":454,"_type":36,"title":455,"_source":38,"_file":456,"_stem":457,"_extension":41},"/shared/en-us/main-navigation","en-us",{"logo":46,"freeTrial":51,"sales":56,"login":61,"items":66,"search":395,"minimal":426,"duo":445},{"config":47},{"href":48,"dataGaName":49,"dataGaLocation":50},"/","gitlab logo","header",{"text":52,"config":53},"Get free trial",{"href":54,"dataGaName":55,"dataGaLocation":50},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com&glm_content=default-saas-trial/","free trial",{"text":57,"config":58},"Talk to sales",{"href":59,"dataGaName":60,"dataGaLocation":50},"/sales/","sales",{"text":62,"config":63},"Sign in",{"href":64,"dataGaName":65,"dataGaLocation":50},"https://gitlab.com/users/sign_in/","sign in",[67,111,207,212,316,376],{"text":68,"config":69,"cards":71,"footer":94},"Platform",{"dataNavLevelOne":70},"platform",[72,78,86],{"title":68,"description":73,"link":74},"The most comprehensive AI-powered DevSecOps Platform",{"text":75,"config":76},"Explore our Platform",{"href":77,"dataGaName":70,"dataGaLocation":50},"/platform/",{"title":79,"description":80,"link":81},"GitLab Duo (AI)","Build software faster with AI at every stage of development",{"text":82,"config":83},"Meet GitLab Duo",{"href":84,"dataGaName":85,"dataGaLocation":50},"/gitlab-duo/","gitlab duo ai",{"title":87,"description":88,"link":89},"Why GitLab","10 reasons why Enterprises choose GitLab",{"text":90,"config":91},"Learn more",{"href":92,"dataGaName":93,"dataGaLocation":50},"/why-gitlab/","why gitlab",{"title":95,"items":96},"Get started with",[97,102,107],{"text":98,"config":99},"Platform Engineering",{"href":100,"dataGaName":101,"dataGaLocation":50},"/solutions/platform-engineering/","platform engineering",{"text":103,"config":104},"Developer Experience",{"href":105,"dataGaName":106,"dataGaLocation":50},"/developer-experience/","Developer experience",{"text":108,"config":109},"MLOps",{"href":110,"dataGaName":108,"dataGaLocation":50},"/topics/devops/the-role-of-ai-in-devops/",{"text":112,"left":113,"config":114,"link":116,"lists":120,"footer":189},"Product",true,{"dataNavLevelOne":115},"solutions",{"text":117,"config":118},"View all Solutions",{"href":119,"dataGaName":115,"dataGaLocation":50},"/solutions/",[121,146,168],{"title":122,"description":123,"link":124,"items":129},"Automation","CI/CD and automation to accelerate deployment",{"config":125},{"icon":126,"href":127,"dataGaName":128,"dataGaLocation":50},"AutomatedCodeAlt","/solutions/delivery-automation/","automated software delivery",[130,134,138,142],{"text":131,"config":132},"CI/CD",{"href":133,"dataGaLocation":50,"dataGaName":131},"/solutions/continuous-integration/",{"text":135,"config":136},"AI-Assisted Development",{"href":84,"dataGaLocation":50,"dataGaName":137},"AI assisted development",{"text":139,"config":140},"Source Code Management",{"href":141,"dataGaLocation":50,"dataGaName":139},"/solutions/source-code-management/",{"text":143,"config":144},"Automated Software Delivery",{"href":127,"dataGaLocation":50,"dataGaName":145},"Automated software delivery",{"title":147,"description":148,"link":149,"items":154},"Security","Deliver code faster without compromising security",{"config":150},{"href":151,"dataGaName":152,"dataGaLocation":50,"icon":153},"/solutions/security-compliance/","security and compliance","ShieldCheckLight",[155,158,163],{"text":156,"config":157},"Security & Compliance",{"href":151,"dataGaLocation":50,"dataGaName":156},{"text":159,"config":160},"Software Supply Chain Security",{"href":161,"dataGaLocation":50,"dataGaName":162},"/solutions/supply-chain/","Software supply chain security",{"text":164,"config":165},"Compliance & Governance",{"href":166,"dataGaLocation":50,"dataGaName":167},"/solutions/continuous-software-compliance/","Compliance and governance",{"title":169,"link":170,"items":175},"Measurement",{"config":171},{"icon":172,"href":173,"dataGaName":174,"dataGaLocation":50},"DigitalTransformation","/solutions/visibility-measurement/","visibility and measurement",[176,180,184],{"text":177,"config":178},"Visibility & Measurement",{"href":173,"dataGaLocation":50,"dataGaName":179},"Visibility and Measurement",{"text":181,"config":182},"Value Stream Management",{"href":183,"dataGaLocation":50,"dataGaName":181},"/solutions/value-stream-management/",{"text":185,"config":186},"Analytics & Insights",{"href":187,"dataGaLocation":50,"dataGaName":188},"/solutions/analytics-and-insights/","Analytics and insights",{"title":190,"items":191},"GitLab for",[192,197,202],{"text":193,"config":194},"Enterprise",{"href":195,"dataGaLocation":50,"dataGaName":196},"/enterprise/","enterprise",{"text":198,"config":199},"Small Business",{"href":200,"dataGaLocation":50,"dataGaName":201},"/small-business/","small business",{"text":203,"config":204},"Public Sector",{"href":205,"dataGaLocation":50,"dataGaName":206},"/solutions/public-sector/","public sector",{"text":208,"config":209},"Pricing",{"href":210,"dataGaName":211,"dataGaLocation":50,"dataNavLevelOne":211},"/pricing/","pricing",{"text":213,"config":214,"link":216,"lists":220,"feature":303},"Resources",{"dataNavLevelOne":215},"resources",{"text":217,"config":218},"View all resources",{"href":219,"dataGaName":215,"dataGaLocation":50},"/resources/",[221,254,276],{"title":222,"items":223},"Getting started",[224,229,234,239,244,249],{"text":225,"config":226},"Install",{"href":227,"dataGaName":228,"dataGaLocation":50},"/install/","install",{"text":230,"config":231},"Quick start guides",{"href":232,"dataGaName":233,"dataGaLocation":50},"/get-started/","quick setup checklists",{"text":235,"config":236},"Learn",{"href":237,"dataGaLocation":50,"dataGaName":238},"https://university.gitlab.com/","learn",{"text":240,"config":241},"Product documentation",{"href":242,"dataGaName":243,"dataGaLocation":50},"https://docs.gitlab.com/","product documentation",{"text":245,"config":246},"Best practice videos",{"href":247,"dataGaName":248,"dataGaLocation":50},"/getting-started-videos/","best practice videos",{"text":250,"config":251},"Integrations",{"href":252,"dataGaName":253,"dataGaLocation":50},"/integrations/","integrations",{"title":255,"items":256},"Discover",[257,262,266,271],{"text":258,"config":259},"Customer success stories",{"href":260,"dataGaName":261,"dataGaLocation":50},"/customers/","customer success stories",{"text":263,"config":264},"Blog",{"href":265,"dataGaName":5,"dataGaLocation":50},"/blog/",{"text":267,"config":268},"Remote",{"href":269,"dataGaName":270,"dataGaLocation":50},"https://handbook.gitlab.com/handbook/company/culture/all-remote/","remote",{"text":272,"config":273},"TeamOps",{"href":274,"dataGaName":275,"dataGaLocation":50},"/teamops/","teamops",{"title":277,"items":278},"Connect",[279,284,288,293,298],{"text":280,"config":281},"GitLab Services",{"href":282,"dataGaName":283,"dataGaLocation":50},"/services/","services",{"text":285,"config":286},"Community",{"href":287,"dataGaName":24,"dataGaLocation":50},"/community/",{"text":289,"config":290},"Forum",{"href":291,"dataGaName":292,"dataGaLocation":50},"https://forum.gitlab.com/","forum",{"text":294,"config":295},"Events",{"href":296,"dataGaName":297,"dataGaLocation":50},"/events/","events",{"text":299,"config":300},"Partners",{"href":301,"dataGaName":302,"dataGaLocation":50},"/partners/","partners",{"backgroundColor":304,"textColor":305,"text":306,"image":307,"link":311},"#2f2a6b","#fff","Insights for the future of software development",{"altText":308,"config":309},"the source promo card",{"src":310},"/images/navigation/the-source-promo-card.svg",{"text":312,"config":313},"Read the latest",{"href":314,"dataGaName":315,"dataGaLocation":50},"/the-source/","the source",{"text":317,"config":318,"lists":320},"Company",{"dataNavLevelOne":319},"company",[321],{"items":322},[323,328,334,336,341,346,351,356,361,366,371],{"text":324,"config":325},"About",{"href":326,"dataGaName":327,"dataGaLocation":50},"/company/","about",{"text":329,"config":330,"footerGa":333},"Jobs",{"href":331,"dataGaName":332,"dataGaLocation":50},"/jobs/","jobs",{"dataGaName":332},{"text":294,"config":335},{"href":296,"dataGaName":297,"dataGaLocation":50},{"text":337,"config":338},"Leadership",{"href":339,"dataGaName":340,"dataGaLocation":50},"/company/team/e-group/","leadership",{"text":342,"config":343},"Team",{"href":344,"dataGaName":345,"dataGaLocation":50},"/company/team/","team",{"text":347,"config":348},"Handbook",{"href":349,"dataGaName":350,"dataGaLocation":50},"https://handbook.gitlab.com/","handbook",{"text":352,"config":353},"Investor relations",{"href":354,"dataGaName":355,"dataGaLocation":50},"https://ir.gitlab.com/","investor relations",{"text":357,"config":358},"Trust Center",{"href":359,"dataGaName":360,"dataGaLocation":50},"/security/","trust center",{"text":362,"config":363},"AI Transparency Center",{"href":364,"dataGaName":365,"dataGaLocation":50},"/ai-transparency-center/","ai transparency center",{"text":367,"config":368},"Newsletter",{"href":369,"dataGaName":370,"dataGaLocation":50},"/company/contact/","newsletter",{"text":372,"config":373},"Press",{"href":374,"dataGaName":375,"dataGaLocation":50},"/press/","press",{"text":377,"config":378,"lists":379},"Contact us",{"dataNavLevelOne":319},[380],{"items":381},[382,385,390],{"text":57,"config":383},{"href":59,"dataGaName":384,"dataGaLocation":50},"talk to sales",{"text":386,"config":387},"Get help",{"href":388,"dataGaName":389,"dataGaLocation":50},"/support/","get help",{"text":391,"config":392},"Customer portal",{"href":393,"dataGaName":394,"dataGaLocation":50},"https://customers.gitlab.com/customers/sign_in/","customer portal",{"close":396,"login":397,"suggestions":404},"Close",{"text":398,"link":399},"To search repositories and projects, login to",{"text":400,"config":401},"gitlab.com",{"href":64,"dataGaName":402,"dataGaLocation":403},"search login","search",{"text":405,"default":406},"Suggestions",[407,409,413,415,419,423],{"text":79,"config":408},{"href":84,"dataGaName":79,"dataGaLocation":403},{"text":410,"config":411},"Code Suggestions (AI)",{"href":412,"dataGaName":410,"dataGaLocation":403},"/solutions/code-suggestions/",{"text":131,"config":414},{"href":133,"dataGaName":131,"dataGaLocation":403},{"text":416,"config":417},"GitLab on AWS",{"href":418,"dataGaName":416,"dataGaLocation":403},"/partners/technology-partners/aws/",{"text":420,"config":421},"GitLab on Google Cloud",{"href":422,"dataGaName":420,"dataGaLocation":403},"/partners/technology-partners/google-cloud-platform/",{"text":424,"config":425},"Why GitLab?",{"href":92,"dataGaName":424,"dataGaLocation":403},{"freeTrial":427,"mobileIcon":432,"desktopIcon":437,"secondaryButton":440},{"text":428,"config":429},"Start free trial",{"href":430,"dataGaName":55,"dataGaLocation":431},"https://gitlab.com/-/trials/new/","nav",{"altText":433,"config":434},"Gitlab Icon",{"src":435,"dataGaName":436,"dataGaLocation":431},"/images/brand/gitlab-logo-tanuki.svg","gitlab icon",{"altText":433,"config":438},{"src":439,"dataGaName":436,"dataGaLocation":431},"/images/brand/gitlab-logo-type.svg",{"text":441,"config":442},"Get Started",{"href":443,"dataGaName":444,"dataGaLocation":431},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com/compare/gitlab-vs-github/","get started",{"freeTrial":446,"mobileIcon":450,"desktopIcon":452},{"text":447,"config":448},"Learn more about GitLab Duo",{"href":84,"dataGaName":449,"dataGaLocation":431},"gitlab duo",{"altText":433,"config":451},{"src":435,"dataGaName":436,"dataGaLocation":431},{"altText":433,"config":453},{"src":439,"dataGaName":436,"dataGaLocation":431},"content:shared:en-us:main-navigation.yml","Main Navigation","shared/en-us/main-navigation.yml","shared/en-us/main-navigation",{"_path":459,"_dir":44,"_draft":6,"_partial":6,"_locale":7,"title":460,"button":461,"image":465,"config":468,"_id":470,"_type":36,"_source":38,"_file":471,"_stem":472,"_extension":41},"/shared/en-us/banner","is now in public beta!",{"text":90,"config":462},{"href":463,"dataGaName":464,"dataGaLocation":50},"/gitlab-duo/agent-platform/","duo banner",{"config":466},{"src":467},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1753720689/somrf9zaunk0xlt7ne4x.svg",{"layout":469},"release","content:shared:en-us:banner.yml","shared/en-us/banner.yml","shared/en-us/banner",{"_path":474,"_dir":44,"_draft":6,"_partial":6,"_locale":7,"data":475,"_id":679,"_type":36,"title":680,"_source":38,"_file":681,"_stem":682,"_extension":41},"/shared/en-us/main-footer",{"text":476,"source":477,"edit":483,"contribute":488,"config":493,"items":498,"minimal":671},"Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license",{"text":478,"config":479},"View page source",{"href":480,"dataGaName":481,"dataGaLocation":482},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/","page source","footer",{"text":484,"config":485},"Edit this page",{"href":486,"dataGaName":487,"dataGaLocation":482},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/content/","web ide",{"text":489,"config":490},"Please contribute",{"href":491,"dataGaName":492,"dataGaLocation":482},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/CONTRIBUTING.md/","please contribute",{"twitter":494,"facebook":495,"youtube":496,"linkedin":497},"https://twitter.com/gitlab","https://www.facebook.com/gitlab","https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg","https://www.linkedin.com/company/gitlab-com",[499,522,578,607,641],{"title":68,"links":500,"subMenu":505},[501],{"text":502,"config":503},"DevSecOps platform",{"href":77,"dataGaName":504,"dataGaLocation":482},"devsecops platform",[506],{"title":208,"links":507},[508,512,517],{"text":509,"config":510},"View plans",{"href":210,"dataGaName":511,"dataGaLocation":482},"view plans",{"text":513,"config":514},"Why Premium?",{"href":515,"dataGaName":516,"dataGaLocation":482},"/pricing/premium/","why premium",{"text":518,"config":519},"Why Ultimate?",{"href":520,"dataGaName":521,"dataGaLocation":482},"/pricing/ultimate/","why ultimate",{"title":523,"links":524},"Solutions",[525,530,533,535,540,545,549,552,556,560,562,565,568,573],{"text":526,"config":527},"Digital transformation",{"href":528,"dataGaName":529,"dataGaLocation":482},"/topics/digital-transformation/","digital transformation",{"text":156,"config":531},{"href":151,"dataGaName":532,"dataGaLocation":482},"security & compliance",{"text":145,"config":534},{"href":127,"dataGaName":128,"dataGaLocation":482},{"text":536,"config":537},"Agile development",{"href":538,"dataGaName":539,"dataGaLocation":482},"/solutions/agile-delivery/","agile delivery",{"text":541,"config":542},"Cloud transformation",{"href":543,"dataGaName":544,"dataGaLocation":482},"/topics/cloud-native/","cloud transformation",{"text":546,"config":547},"SCM",{"href":141,"dataGaName":548,"dataGaLocation":482},"source code management",{"text":131,"config":550},{"href":133,"dataGaName":551,"dataGaLocation":482},"continuous integration & delivery",{"text":553,"config":554},"Value stream management",{"href":183,"dataGaName":555,"dataGaLocation":482},"value stream management",{"text":26,"config":557},{"href":558,"dataGaName":559,"dataGaLocation":482},"/solutions/gitops/","gitops",{"text":193,"config":561},{"href":195,"dataGaName":196,"dataGaLocation":482},{"text":563,"config":564},"Small business",{"href":200,"dataGaName":201,"dataGaLocation":482},{"text":566,"config":567},"Public sector",{"href":205,"dataGaName":206,"dataGaLocation":482},{"text":569,"config":570},"Education",{"href":571,"dataGaName":572,"dataGaLocation":482},"/solutions/education/","education",{"text":574,"config":575},"Financial services",{"href":576,"dataGaName":577,"dataGaLocation":482},"/solutions/finance/","financial services",{"title":213,"links":579},[580,582,584,586,589,591,593,595,597,599,601,603,605],{"text":225,"config":581},{"href":227,"dataGaName":228,"dataGaLocation":482},{"text":230,"config":583},{"href":232,"dataGaName":233,"dataGaLocation":482},{"text":235,"config":585},{"href":237,"dataGaName":238,"dataGaLocation":482},{"text":240,"config":587},{"href":242,"dataGaName":588,"dataGaLocation":482},"docs",{"text":263,"config":590},{"href":265,"dataGaName":5,"dataGaLocation":482},{"text":258,"config":592},{"href":260,"dataGaName":261,"dataGaLocation":482},{"text":267,"config":594},{"href":269,"dataGaName":270,"dataGaLocation":482},{"text":280,"config":596},{"href":282,"dataGaName":283,"dataGaLocation":482},{"text":272,"config":598},{"href":274,"dataGaName":275,"dataGaLocation":482},{"text":285,"config":600},{"href":287,"dataGaName":24,"dataGaLocation":482},{"text":289,"config":602},{"href":291,"dataGaName":292,"dataGaLocation":482},{"text":294,"config":604},{"href":296,"dataGaName":297,"dataGaLocation":482},{"text":299,"config":606},{"href":301,"dataGaName":302,"dataGaLocation":482},{"title":317,"links":608},[609,611,613,615,617,619,621,625,630,632,634,636],{"text":324,"config":610},{"href":326,"dataGaName":319,"dataGaLocation":482},{"text":329,"config":612},{"href":331,"dataGaName":332,"dataGaLocation":482},{"text":337,"config":614},{"href":339,"dataGaName":340,"dataGaLocation":482},{"text":342,"config":616},{"href":344,"dataGaName":345,"dataGaLocation":482},{"text":347,"config":618},{"href":349,"dataGaName":350,"dataGaLocation":482},{"text":352,"config":620},{"href":354,"dataGaName":355,"dataGaLocation":482},{"text":622,"config":623},"Sustainability",{"href":624,"dataGaName":622,"dataGaLocation":482},"/sustainability/",{"text":626,"config":627},"Diversity, inclusion and belonging (DIB)",{"href":628,"dataGaName":629,"dataGaLocation":482},"/diversity-inclusion-belonging/","Diversity, inclusion and belonging",{"text":357,"config":631},{"href":359,"dataGaName":360,"dataGaLocation":482},{"text":367,"config":633},{"href":369,"dataGaName":370,"dataGaLocation":482},{"text":372,"config":635},{"href":374,"dataGaName":375,"dataGaLocation":482},{"text":637,"config":638},"Modern Slavery Transparency Statement",{"href":639,"dataGaName":640,"dataGaLocation":482},"https://handbook.gitlab.com/handbook/legal/modern-slavery-act-transparency-statement/","modern slavery transparency statement",{"title":642,"links":643},"Contact Us",[644,647,649,651,656,661,666],{"text":645,"config":646},"Contact an expert",{"href":59,"dataGaName":60,"dataGaLocation":482},{"text":386,"config":648},{"href":388,"dataGaName":389,"dataGaLocation":482},{"text":391,"config":650},{"href":393,"dataGaName":394,"dataGaLocation":482},{"text":652,"config":653},"Status",{"href":654,"dataGaName":655,"dataGaLocation":482},"https://status.gitlab.com/","status",{"text":657,"config":658},"Terms of use",{"href":659,"dataGaName":660,"dataGaLocation":482},"/terms/","terms of use",{"text":662,"config":663},"Privacy statement",{"href":664,"dataGaName":665,"dataGaLocation":482},"/privacy/","privacy statement",{"text":667,"config":668},"Cookie preferences",{"dataGaName":669,"dataGaLocation":482,"id":670,"isOneTrustButton":113},"cookie preferences","ot-sdk-btn",{"items":672},[673,675,677],{"text":657,"config":674},{"href":659,"dataGaName":660,"dataGaLocation":482},{"text":662,"config":676},{"href":664,"dataGaName":665,"dataGaLocation":482},{"text":667,"config":678},{"dataGaName":669,"dataGaLocation":482,"id":670,"isOneTrustButton":113},"content:shared:en-us:main-footer.yml","Main Footer","shared/en-us/main-footer.yml","shared/en-us/main-footer",[684],{"_path":685,"_dir":686,"_draft":6,"_partial":6,"_locale":7,"content":687,"config":691,"_id":693,"_type":36,"title":19,"_source":38,"_file":694,"_stem":695,"_extension":41},"/en-us/blog/authors/stan-hu","authors",{"name":19,"config":688},{"headshot":689,"ctfId":690},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1749659504/Blog/Author%20Headshots/stanhu-headshot.jpg","stanhu",{"template":692},"BlogAuthor","content:en-us:blog:authors:stan-hu.yml","en-us/blog/authors/stan-hu.yml","en-us/blog/authors/stan-hu",{"_path":697,"_dir":44,"_draft":6,"_partial":6,"_locale":7,"header":698,"eyebrow":699,"blurb":700,"button":701,"secondaryButton":705,"_id":707,"_type":36,"title":708,"_source":38,"_file":709,"_stem":710,"_extension":41},"/shared/en-us/next-steps","Start shipping better software faster","50%+ of the Fortune 100 trust GitLab","See what your team can do with the intelligent\n\n\nDevSecOps platform.\n",{"text":52,"config":702},{"href":703,"dataGaName":55,"dataGaLocation":704},"https://gitlab.com/-/trial_registrations/new?glm_content=default-saas-trial&glm_source=about.gitlab.com/","feature",{"text":57,"config":706},{"href":59,"dataGaName":60,"dataGaLocation":704},"content:shared:en-us:next-steps.yml","Next Steps","shared/en-us/next-steps.yml","shared/en-us/next-steps",1754424533429]