[{"data":1,"prerenderedAt":709},["ShallowReactive",2],{"/en-us/blog/developing-gitlab-duo-how-we-validate-and-test-ai-models-at-scale/":3,"navigation-en-us":39,"banner-en-us":455,"footer-en-us":470,"Susie Bitters":680,"next-steps-en-us":694},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"seo":8,"content":16,"config":28,"_id":32,"_type":33,"title":34,"_source":35,"_file":36,"_stem":37,"_extension":38},"/en-us/blog/developing-gitlab-duo-how-we-validate-and-test-ai-models-at-scale","blog",false,"",{"title":9,"description":10,"ogTitle":9,"ogDescription":10,"noIndex":6,"ogImage":11,"ogUrl":12,"ogSiteName":13,"ogType":14,"canonicalUrls":12,"schema":15},"Developing GitLab Duo: How we validate and test AI models at scale","Our blog series debuts with a behind-the-scenes look at how we evaluate LLMs, match them to use cases, and fine-tune them to produce better responses for users.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1749659856/Blog/Hero%20Images/blog-hero-banner-1-0178-820x470-fy25.png","https://about.gitlab.com/blog/developing-gitlab-duo-how-we-validate-and-test-ai-models-at-scale","https://about.gitlab.com","article","\n                        {\n        \"@context\": \"https://schema.org\",\n        \"@type\": \"Article\",\n        \"headline\": \"Developing GitLab Duo: How we validate and test AI models at scale\",\n        \"author\": [{\"@type\":\"Person\",\"name\":\"Susie Bitters\"}],\n        \"datePublished\": \"2024-05-09\",\n      }",{"title":9,"description":10,"authors":17,"heroImage":11,"date":19,"body":20,"category":21,"tags":22},[18],"Susie Bitters","2024-05-09","**_Generative AI marks a monumental shift in the software development industry, making it easier to develop, secure, and operate software. Our new blog series, written by our product and engineering teams, gives you an inside look at how we create, test, and deploy the AI features you need integrated throughout the enterprise. Get to know new capabilities within GitLab Duo and how they will help DevSecOps teams deliver better results for customers._**\n\nGitLab values the trust our customers place in us. Part of maintaining that trust is transparency in how we build, evaluate, and ensure the high-quality functionality of our [GitLab Duo](https://about.gitlab.com/gitlab-duo/) AI features. GitLab Duo features are powered by a diverse set of models, which allows us to support a broad set of use cases and gives our customers flexibility. GitLab is not tied to a single model provider by design. We currently use foundation models from [Google](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/blob/main/ai_gateway/models/vertex_text.py?ref_type=heads#L86) and [Anthropic](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/blob/main/ai_gateway/models/anthropic.py?ref_type=heads#L62). However, we continuously assess what models are the right matches for GitLab Duo’s use cases. In this article, we give you an inside look at our AI model validation process.\n\n> Discover the future of AI-driven software development with our GitLab 17 virtual launch event. [Watch today!](https://about.gitlab.com/seventeen/)\n\n## Understanding LLMs\n\nLarge language models (LLMs) are generative AI models that power many AI features across the platform. Trained on vast datasets, LLMs predict the next word in a sequence based on preceding context. Given an input prompt, they generate human-like text by sampling from the probability distribution of words conditioned on the prompt.\n\nLLMs enable intelligent code suggestions, conversational chatbots, code explanations, vulnerability analysis, and more. Their ability to produce diverse outputs for a given prompt makes standardized quality evaluation challenging. LLMs can be optimized for different characteristics, which is why there are so many AI models actively being developed.\n\n## Testing at scale\n\nUnlike traditional software systems where inputs and outputs can be more easily defined and tested, LLMs produce outputs that are often nuanced, diverse, and context-dependent. Testing these models requires comprehensive strategies that account for subjective and variable interpretations of quality, as well as the stochastic nature of their outputs. We, therefore, cannot judge the quality of an LLM’s output in an individual or anecdotal fashion; instead, we need to be able to examine the overall pattern of an LLM's behavior. To get a sense of those patterns, we need to test at scale. Testing at scale refers to the process of evaluating the performance, reliability, and robustness of a system or application across a large and diverse array of datasets and use cases. Our [Centralized Evaluation Framework (CEF)](https://about.gitlab.com/direction/ai-powered/ai_framework/ai_evaluation/) utilizes thousands of prompts tied to dozens of use cases to allow us to identify significant patterns and assess the overall behavior of our foundational LLMs and the GitLab Duo features in which they are integrated.\n\nTesting at scale helps us:\n\n- **Ensure quality:** Testing at scale enables us to assess the quality and reliability of these models across a wide range of scenarios and inputs. By validating the outputs of these models at scale, we can start to identify patterns and mitigate potential issues such as systematic biases, anomalies, and inaccuracies. \n- **Optimize performance:** Scaling up testing efforts allows GitLab to evaluate the performance and efficiency of LLMs under real-world conditions. This includes assessing factors such as output quality, latency, and cost to optimize the deployment and operation of these models in GitLab Duo features.\n- **Mitigate risk:** Testing LLMs at scale helps mitigate the risks associated with deploying LLMs in critical applications. By conducting thorough testing across diverse datasets and use cases, we can identify and address potential failure modes, security vulnerabilities, and ethical concerns before they impact our customers.\n\nTesting LLMs at scale is imperative for ensuring their reliability and robustness for deployment within the GitLab platform. By investing in comprehensive testing strategies that encompass diverse datasets, use cases, and scenarios, GitLab is working to unlock the full potential of AI-powered workflows while mitigating potential risks.\n\n### How we test at scale\n\nThese are the steps we take to test LLMs at scale.\n\n#### Step 1: Create a prompt library as a proxy for production\nWhile other companies view and use customer data to train their AI features, GitLab currently does not.  As a result, we needed to develop a comprehensive prompt library that is a proxy for both the scale and activity of production.\n\nThis prompt library is composed of questions and answers. The questions represent the kinds of queries or inputs that we would expect to see in production, while the answers represent a ground truth of what our ideal answer would be. This ground truth answer could also be mentally framed as a target answer. Both the question and the answer may be human generated, but are not necessarily so. These question/answer pairs give us a basis for comparison and a reference frame that allow us to tease out differences between models and features. When multiple models are asked the same question and generate different responses, we can use our ground truth answer to determine which model has provided an answer that is most closely aligned to our target and score them accordingly.\n\nAgain, a key element of a comprehensive prompt library is ensuring that it is representative of the inputs that we expect to see in production. We want to know how well foundational models fit to our specific use case, and how well our features are performing. There are numerous benchmark prompt datasets, but those datasets may not be reflective of the use cases that we see for features at GitLab. Our prompt library is designed to be specific to GitLab features and use cases.\n\n#### Step 2: Baseline model performance\n\nOnce we have crafted a prompt library that accurately reflects production activity, we feed those questions into [various models](https://about.gitlab.com/direction/ai-powered/ai_framework/ai_evaluation/foundation_models/) to test how well they serve our customer’s needs. We compare each response to our ground truth and provide it a ranking based on a series of metrics including: [Cosine Similarity Score](https://about.gitlab.com/direction/ai-powered/ai_framework/ai_evaluation/metrics/#similarity-scores), [Cross Similarity Score](https://about.gitlab.com/direction/ai-powered/ai_framework/ai_evaluation/metrics/#cross-similarity-score),  [LLM Judge](https://about.gitlab.com/direction/ai-powered/ai_framework/ai_evaluation/metrics/#llm-judge), and [Consensus Filtering with an LLM Judge](https://about.gitlab.com/direction/ai-powered/ai_framework/ai_evaluation/metrics/#consensus-filtering-with-llm-judge). This first iteration provides us a baseline for how well each model is performing, and guides our selection of a foundational model for our features. For brevity, we won’t go into the details here, but we encourage you to [learn more about more about the metrics here](https://about.gitlab.com/direction/ai-powered/ai_framework/ai_evaluation/metrics/). It is important to note this isn’t a solved problem; the wider AI industry is actively researching and developing new techniques. GitLab’s model validation team keeps a pulse on the industry and is continuously iterating on how we measure and score the LLMs GitLab Duo uses.  \n\n#### Step 3: Feature development\n\nNow that we have a baseline for our selected model's performance, we can start developing our features with confidence. While prompt engineering gets a lot of buzz, focusing entirely on changing the behavior of a model via prompting (or any other technique) without validation means that you are operating in the dark and very possibly overfitting your prompting. You may solve one problem, but be causing a dozen more. You would never know. Creating a baseline for a model's performance allows us to track how we are changing behavior over time for all our necessary use cases. At GitLab, we re-validate the performance of our features on a daily basis during active development to help ensure that all changes improve the overall functionality.\n\n#### Step 4: Iterate, iterate, iterate\n\nHere is how our experimental iterations work. Each cycle, we examine the scores from our tests at scale to identify patterns:\n\n- What are the commonalities across our weakest areas?\n- Is our feature performing poorly based on a specific metric or on a certain use case?\n- Do we see consistent errors popping up in response to a certain kind of question?\n\nOnly when we test at scale do these kinds of patterns begin to emerge and allow us to focus our experiments. Based on these patterns, we propose a variety of experiments or approaches to try to improve performance in a specific area and on a specific metric.\n\nHowever, testing at scale is both expensive and time-consuming. To enable faster and less expensive iteration, we craft a smaller scale dataset to act as a mini-proxy. The focused subset will be weighted to include question/answer pairs that we know we want to improve upon, and the broader subset will also include sampling of all the other use cases and scores to ensure that our changes aren't adversely affecting the feature broadly. Make your change and run it against the focused subset of data. How does the new response compare to the baseline? How does it compare to the ground truth?\n\nOnce we have found a prompt that addresses the specific use case we are working on with the focused subset, we validate that prompt against a broader subset of data to help ensure that it won’t adversely affect other areas of the feature. Only when we believe that the new prompt improves our performance in our target area through validation metrics AND doesn’t degrade performance elsewhere, do we push that change to production.\n\nThe entire Centralized Evaluation Framework is then run against the new prompt and we validate that it has increased the performance of the entire feature against the baseline from the day before. In this way, GitLab is constantly iterating to help ensure that you are getting the latest and greatest performance of AI-powered features across the GitLab ecosystem. This allows us to ensure that we keep working faster, together.\n\n### Making GitLab Duo even better\n\nHopefully this gives you insight into how we’re responsibly developing GitLab Duo features. This process has been developed as we’ve brought [GitLab Duo Code Suggestions](https://docs.gitlab.com/ee/user/project/repository/code_suggestions/) and [GitLab Duo Chat](https://docs.gitlab.com/ee/user/gitlab_duo_chat.html) to general availability. We’ve also integrated this validation process into our development process as we iterate on GitLab Duo features. It’s a lot of trial and error, and many times fixing one thing breaks three others. But we have data-driven insights into those impacts, which helps us ensure that GitLab Duo is always getting better.\n\n> Start a [free trial of GitLab Duo](https://about.gitlab.com/gitlab-duo/#free-trial) today!\n\n ## Resources\n - [GitLab AI Transparency Center](https://about.gitlab.com/ai-transparency-center/)\n - [GitLab's AI Ethics Principles for Product Development](https://handbook.gitlab.com/handbook/legal/ethics-compliance-program/ai-ethics-principles/)\n - [GitLab AI-powered Direction page](https://about.gitlab.com/direction/ai-powered/)\n\n\u003Cfigure class=video_container>\n\u003Ciframe width=560 height=315 src=\"https://www.youtube-nocookie.com/embed/LifJdU3Qagw?si=A4kl6d32wPYC4168\" title=\"YouTube video player\" frameborder=0 allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen=\"\">\u003C/iframe>\n\u003C/figure>\n\n## Read more of the \"Developing GitLab Duo\" series\n\n- [Developing GitLab Duo: AI Impact analytics dashboard measures the ROI of AI](https://about.gitlab.com/blog/developing-gitlab-duo-ai-impact-analytics-dashboard-measures-the-roi-of-ai/)\n- [Developing GitLab Duo: How we are dogfooding our AI features](https://about.gitlab.com/blog/developing-gitlab-duo-how-we-are-dogfooding-our-ai-features/) \n- [Developing GitLab Duo: Secure and thoroughly test AI-generated code](https://about.gitlab.com/blog/how-gitlab-duo-helps-secure-and-thoroughly-test-ai-generated-code/)\n- [Developing GitLab Duo: Blending AI and Root Cause Analysis to fix CI/CD pipelines](https://about.gitlab.com/blog/developing-gitlab-duo-blending-ai-and-root-cause-analysis-to-fix-ci-cd/)","ai-ml",[23,24,25,26,27],"AI/ML","DevSecOps","DevSecOps platform","features","inside GitLab",{"slug":29,"featured":30,"template":31},"developing-gitlab-duo-how-we-validate-and-test-ai-models-at-scale",true,"BlogPost","content:en-us:blog:developing-gitlab-duo-how-we-validate-and-test-ai-models-at-scale.yml","yaml","Developing Gitlab Duo How We Validate And Test Ai Models At Scale","content","en-us/blog/developing-gitlab-duo-how-we-validate-and-test-ai-models-at-scale.yml","en-us/blog/developing-gitlab-duo-how-we-validate-and-test-ai-models-at-scale","yml",{"_path":40,"_dir":41,"_draft":6,"_partial":6,"_locale":7,"data":42,"_id":451,"_type":33,"title":452,"_source":35,"_file":453,"_stem":454,"_extension":38},"/shared/en-us/main-navigation","en-us",{"logo":43,"freeTrial":48,"sales":53,"login":58,"items":63,"search":392,"minimal":423,"duo":442},{"config":44},{"href":45,"dataGaName":46,"dataGaLocation":47},"/","gitlab logo","header",{"text":49,"config":50},"Get free trial",{"href":51,"dataGaName":52,"dataGaLocation":47},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com&glm_content=default-saas-trial/","free trial",{"text":54,"config":55},"Talk to sales",{"href":56,"dataGaName":57,"dataGaLocation":47},"/sales/","sales",{"text":59,"config":60},"Sign in",{"href":61,"dataGaName":62,"dataGaLocation":47},"https://gitlab.com/users/sign_in/","sign in",[64,108,203,208,313,373],{"text":65,"config":66,"cards":68,"footer":91},"Platform",{"dataNavLevelOne":67},"platform",[69,75,83],{"title":65,"description":70,"link":71},"The most comprehensive AI-powered DevSecOps Platform",{"text":72,"config":73},"Explore our Platform",{"href":74,"dataGaName":67,"dataGaLocation":47},"/platform/",{"title":76,"description":77,"link":78},"GitLab Duo (AI)","Build software faster with AI at every stage of development",{"text":79,"config":80},"Meet GitLab Duo",{"href":81,"dataGaName":82,"dataGaLocation":47},"/gitlab-duo/","gitlab duo ai",{"title":84,"description":85,"link":86},"Why GitLab","10 reasons why Enterprises choose GitLab",{"text":87,"config":88},"Learn more",{"href":89,"dataGaName":90,"dataGaLocation":47},"/why-gitlab/","why gitlab",{"title":92,"items":93},"Get started with",[94,99,104],{"text":95,"config":96},"Platform Engineering",{"href":97,"dataGaName":98,"dataGaLocation":47},"/solutions/platform-engineering/","platform engineering",{"text":100,"config":101},"Developer Experience",{"href":102,"dataGaName":103,"dataGaLocation":47},"/developer-experience/","Developer experience",{"text":105,"config":106},"MLOps",{"href":107,"dataGaName":105,"dataGaLocation":47},"/topics/devops/the-role-of-ai-in-devops/",{"text":109,"left":30,"config":110,"link":112,"lists":116,"footer":185},"Product",{"dataNavLevelOne":111},"solutions",{"text":113,"config":114},"View all Solutions",{"href":115,"dataGaName":111,"dataGaLocation":47},"/solutions/",[117,142,164],{"title":118,"description":119,"link":120,"items":125},"Automation","CI/CD and automation to accelerate deployment",{"config":121},{"icon":122,"href":123,"dataGaName":124,"dataGaLocation":47},"AutomatedCodeAlt","/solutions/delivery-automation/","automated software delivery",[126,130,134,138],{"text":127,"config":128},"CI/CD",{"href":129,"dataGaLocation":47,"dataGaName":127},"/solutions/continuous-integration/",{"text":131,"config":132},"AI-Assisted Development",{"href":81,"dataGaLocation":47,"dataGaName":133},"AI assisted development",{"text":135,"config":136},"Source Code Management",{"href":137,"dataGaLocation":47,"dataGaName":135},"/solutions/source-code-management/",{"text":139,"config":140},"Automated Software Delivery",{"href":123,"dataGaLocation":47,"dataGaName":141},"Automated software delivery",{"title":143,"description":144,"link":145,"items":150},"Security","Deliver code faster without compromising security",{"config":146},{"href":147,"dataGaName":148,"dataGaLocation":47,"icon":149},"/solutions/security-compliance/","security and compliance","ShieldCheckLight",[151,154,159],{"text":152,"config":153},"Security & Compliance",{"href":147,"dataGaLocation":47,"dataGaName":152},{"text":155,"config":156},"Software Supply Chain Security",{"href":157,"dataGaLocation":47,"dataGaName":158},"/solutions/supply-chain/","Software supply chain security",{"text":160,"config":161},"Compliance & Governance",{"href":162,"dataGaLocation":47,"dataGaName":163},"/solutions/continuous-software-compliance/","Compliance and governance",{"title":165,"link":166,"items":171},"Measurement",{"config":167},{"icon":168,"href":169,"dataGaName":170,"dataGaLocation":47},"DigitalTransformation","/solutions/visibility-measurement/","visibility and measurement",[172,176,180],{"text":173,"config":174},"Visibility & Measurement",{"href":169,"dataGaLocation":47,"dataGaName":175},"Visibility and Measurement",{"text":177,"config":178},"Value Stream Management",{"href":179,"dataGaLocation":47,"dataGaName":177},"/solutions/value-stream-management/",{"text":181,"config":182},"Analytics & Insights",{"href":183,"dataGaLocation":47,"dataGaName":184},"/solutions/analytics-and-insights/","Analytics and insights",{"title":186,"items":187},"GitLab for",[188,193,198],{"text":189,"config":190},"Enterprise",{"href":191,"dataGaLocation":47,"dataGaName":192},"/enterprise/","enterprise",{"text":194,"config":195},"Small Business",{"href":196,"dataGaLocation":47,"dataGaName":197},"/small-business/","small business",{"text":199,"config":200},"Public Sector",{"href":201,"dataGaLocation":47,"dataGaName":202},"/solutions/public-sector/","public sector",{"text":204,"config":205},"Pricing",{"href":206,"dataGaName":207,"dataGaLocation":47,"dataNavLevelOne":207},"/pricing/","pricing",{"text":209,"config":210,"link":212,"lists":216,"feature":300},"Resources",{"dataNavLevelOne":211},"resources",{"text":213,"config":214},"View all resources",{"href":215,"dataGaName":211,"dataGaLocation":47},"/resources/",[217,250,272],{"title":218,"items":219},"Getting started",[220,225,230,235,240,245],{"text":221,"config":222},"Install",{"href":223,"dataGaName":224,"dataGaLocation":47},"/install/","install",{"text":226,"config":227},"Quick start guides",{"href":228,"dataGaName":229,"dataGaLocation":47},"/get-started/","quick setup checklists",{"text":231,"config":232},"Learn",{"href":233,"dataGaLocation":47,"dataGaName":234},"https://university.gitlab.com/","learn",{"text":236,"config":237},"Product documentation",{"href":238,"dataGaName":239,"dataGaLocation":47},"https://docs.gitlab.com/","product documentation",{"text":241,"config":242},"Best practice videos",{"href":243,"dataGaName":244,"dataGaLocation":47},"/getting-started-videos/","best practice videos",{"text":246,"config":247},"Integrations",{"href":248,"dataGaName":249,"dataGaLocation":47},"/integrations/","integrations",{"title":251,"items":252},"Discover",[253,258,262,267],{"text":254,"config":255},"Customer success stories",{"href":256,"dataGaName":257,"dataGaLocation":47},"/customers/","customer success stories",{"text":259,"config":260},"Blog",{"href":261,"dataGaName":5,"dataGaLocation":47},"/blog/",{"text":263,"config":264},"Remote",{"href":265,"dataGaName":266,"dataGaLocation":47},"https://handbook.gitlab.com/handbook/company/culture/all-remote/","remote",{"text":268,"config":269},"TeamOps",{"href":270,"dataGaName":271,"dataGaLocation":47},"/teamops/","teamops",{"title":273,"items":274},"Connect",[275,280,285,290,295],{"text":276,"config":277},"GitLab Services",{"href":278,"dataGaName":279,"dataGaLocation":47},"/services/","services",{"text":281,"config":282},"Community",{"href":283,"dataGaName":284,"dataGaLocation":47},"/community/","community",{"text":286,"config":287},"Forum",{"href":288,"dataGaName":289,"dataGaLocation":47},"https://forum.gitlab.com/","forum",{"text":291,"config":292},"Events",{"href":293,"dataGaName":294,"dataGaLocation":47},"/events/","events",{"text":296,"config":297},"Partners",{"href":298,"dataGaName":299,"dataGaLocation":47},"/partners/","partners",{"backgroundColor":301,"textColor":302,"text":303,"image":304,"link":308},"#2f2a6b","#fff","Insights for the future of software development",{"altText":305,"config":306},"the source promo card",{"src":307},"/images/navigation/the-source-promo-card.svg",{"text":309,"config":310},"Read the latest",{"href":311,"dataGaName":312,"dataGaLocation":47},"/the-source/","the source",{"text":314,"config":315,"lists":317},"Company",{"dataNavLevelOne":316},"company",[318],{"items":319},[320,325,331,333,338,343,348,353,358,363,368],{"text":321,"config":322},"About",{"href":323,"dataGaName":324,"dataGaLocation":47},"/company/","about",{"text":326,"config":327,"footerGa":330},"Jobs",{"href":328,"dataGaName":329,"dataGaLocation":47},"/jobs/","jobs",{"dataGaName":329},{"text":291,"config":332},{"href":293,"dataGaName":294,"dataGaLocation":47},{"text":334,"config":335},"Leadership",{"href":336,"dataGaName":337,"dataGaLocation":47},"/company/team/e-group/","leadership",{"text":339,"config":340},"Team",{"href":341,"dataGaName":342,"dataGaLocation":47},"/company/team/","team",{"text":344,"config":345},"Handbook",{"href":346,"dataGaName":347,"dataGaLocation":47},"https://handbook.gitlab.com/","handbook",{"text":349,"config":350},"Investor relations",{"href":351,"dataGaName":352,"dataGaLocation":47},"https://ir.gitlab.com/","investor relations",{"text":354,"config":355},"Trust Center",{"href":356,"dataGaName":357,"dataGaLocation":47},"/security/","trust center",{"text":359,"config":360},"AI Transparency Center",{"href":361,"dataGaName":362,"dataGaLocation":47},"/ai-transparency-center/","ai transparency center",{"text":364,"config":365},"Newsletter",{"href":366,"dataGaName":367,"dataGaLocation":47},"/company/contact/","newsletter",{"text":369,"config":370},"Press",{"href":371,"dataGaName":372,"dataGaLocation":47},"/press/","press",{"text":374,"config":375,"lists":376},"Contact us",{"dataNavLevelOne":316},[377],{"items":378},[379,382,387],{"text":54,"config":380},{"href":56,"dataGaName":381,"dataGaLocation":47},"talk to sales",{"text":383,"config":384},"Get help",{"href":385,"dataGaName":386,"dataGaLocation":47},"/support/","get help",{"text":388,"config":389},"Customer portal",{"href":390,"dataGaName":391,"dataGaLocation":47},"https://customers.gitlab.com/customers/sign_in/","customer portal",{"close":393,"login":394,"suggestions":401},"Close",{"text":395,"link":396},"To search repositories and projects, login to",{"text":397,"config":398},"gitlab.com",{"href":61,"dataGaName":399,"dataGaLocation":400},"search login","search",{"text":402,"default":403},"Suggestions",[404,406,410,412,416,420],{"text":76,"config":405},{"href":81,"dataGaName":76,"dataGaLocation":400},{"text":407,"config":408},"Code Suggestions (AI)",{"href":409,"dataGaName":407,"dataGaLocation":400},"/solutions/code-suggestions/",{"text":127,"config":411},{"href":129,"dataGaName":127,"dataGaLocation":400},{"text":413,"config":414},"GitLab on AWS",{"href":415,"dataGaName":413,"dataGaLocation":400},"/partners/technology-partners/aws/",{"text":417,"config":418},"GitLab on Google Cloud",{"href":419,"dataGaName":417,"dataGaLocation":400},"/partners/technology-partners/google-cloud-platform/",{"text":421,"config":422},"Why GitLab?",{"href":89,"dataGaName":421,"dataGaLocation":400},{"freeTrial":424,"mobileIcon":429,"desktopIcon":434,"secondaryButton":437},{"text":425,"config":426},"Start free trial",{"href":427,"dataGaName":52,"dataGaLocation":428},"https://gitlab.com/-/trials/new/","nav",{"altText":430,"config":431},"Gitlab Icon",{"src":432,"dataGaName":433,"dataGaLocation":428},"/images/brand/gitlab-logo-tanuki.svg","gitlab icon",{"altText":430,"config":435},{"src":436,"dataGaName":433,"dataGaLocation":428},"/images/brand/gitlab-logo-type.svg",{"text":438,"config":439},"Get Started",{"href":440,"dataGaName":441,"dataGaLocation":428},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com/compare/gitlab-vs-github/","get started",{"freeTrial":443,"mobileIcon":447,"desktopIcon":449},{"text":444,"config":445},"Learn more about GitLab Duo",{"href":81,"dataGaName":446,"dataGaLocation":428},"gitlab duo",{"altText":430,"config":448},{"src":432,"dataGaName":433,"dataGaLocation":428},{"altText":430,"config":450},{"src":436,"dataGaName":433,"dataGaLocation":428},"content:shared:en-us:main-navigation.yml","Main Navigation","shared/en-us/main-navigation.yml","shared/en-us/main-navigation",{"_path":456,"_dir":41,"_draft":6,"_partial":6,"_locale":7,"title":457,"button":458,"image":462,"config":465,"_id":467,"_type":33,"_source":35,"_file":468,"_stem":469,"_extension":38},"/shared/en-us/banner","is now in public beta!",{"text":87,"config":459},{"href":460,"dataGaName":461,"dataGaLocation":47},"/gitlab-duo/agent-platform/","duo banner",{"config":463},{"src":464},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1753720689/somrf9zaunk0xlt7ne4x.svg",{"layout":466},"release","content:shared:en-us:banner.yml","shared/en-us/banner.yml","shared/en-us/banner",{"_path":471,"_dir":41,"_draft":6,"_partial":6,"_locale":7,"data":472,"_id":676,"_type":33,"title":677,"_source":35,"_file":678,"_stem":679,"_extension":38},"/shared/en-us/main-footer",{"text":473,"source":474,"edit":480,"contribute":485,"config":490,"items":495,"minimal":668},"Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license",{"text":475,"config":476},"View page source",{"href":477,"dataGaName":478,"dataGaLocation":479},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/","page source","footer",{"text":481,"config":482},"Edit this page",{"href":483,"dataGaName":484,"dataGaLocation":479},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/content/","web ide",{"text":486,"config":487},"Please contribute",{"href":488,"dataGaName":489,"dataGaLocation":479},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/CONTRIBUTING.md/","please contribute",{"twitter":491,"facebook":492,"youtube":493,"linkedin":494},"https://twitter.com/gitlab","https://www.facebook.com/gitlab","https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg","https://www.linkedin.com/company/gitlab-com",[496,518,575,604,638],{"title":65,"links":497,"subMenu":501},[498],{"text":25,"config":499},{"href":74,"dataGaName":500,"dataGaLocation":479},"devsecops platform",[502],{"title":204,"links":503},[504,508,513],{"text":505,"config":506},"View plans",{"href":206,"dataGaName":507,"dataGaLocation":479},"view plans",{"text":509,"config":510},"Why Premium?",{"href":511,"dataGaName":512,"dataGaLocation":479},"/pricing/premium/","why premium",{"text":514,"config":515},"Why Ultimate?",{"href":516,"dataGaName":517,"dataGaLocation":479},"/pricing/ultimate/","why ultimate",{"title":519,"links":520},"Solutions",[521,526,529,531,536,541,545,548,552,557,559,562,565,570],{"text":522,"config":523},"Digital transformation",{"href":524,"dataGaName":525,"dataGaLocation":479},"/topics/digital-transformation/","digital transformation",{"text":152,"config":527},{"href":147,"dataGaName":528,"dataGaLocation":479},"security & compliance",{"text":141,"config":530},{"href":123,"dataGaName":124,"dataGaLocation":479},{"text":532,"config":533},"Agile development",{"href":534,"dataGaName":535,"dataGaLocation":479},"/solutions/agile-delivery/","agile delivery",{"text":537,"config":538},"Cloud transformation",{"href":539,"dataGaName":540,"dataGaLocation":479},"/topics/cloud-native/","cloud transformation",{"text":542,"config":543},"SCM",{"href":137,"dataGaName":544,"dataGaLocation":479},"source code management",{"text":127,"config":546},{"href":129,"dataGaName":547,"dataGaLocation":479},"continuous integration & delivery",{"text":549,"config":550},"Value stream management",{"href":179,"dataGaName":551,"dataGaLocation":479},"value stream management",{"text":553,"config":554},"GitOps",{"href":555,"dataGaName":556,"dataGaLocation":479},"/solutions/gitops/","gitops",{"text":189,"config":558},{"href":191,"dataGaName":192,"dataGaLocation":479},{"text":560,"config":561},"Small business",{"href":196,"dataGaName":197,"dataGaLocation":479},{"text":563,"config":564},"Public sector",{"href":201,"dataGaName":202,"dataGaLocation":479},{"text":566,"config":567},"Education",{"href":568,"dataGaName":569,"dataGaLocation":479},"/solutions/education/","education",{"text":571,"config":572},"Financial services",{"href":573,"dataGaName":574,"dataGaLocation":479},"/solutions/finance/","financial services",{"title":209,"links":576},[577,579,581,583,586,588,590,592,594,596,598,600,602],{"text":221,"config":578},{"href":223,"dataGaName":224,"dataGaLocation":479},{"text":226,"config":580},{"href":228,"dataGaName":229,"dataGaLocation":479},{"text":231,"config":582},{"href":233,"dataGaName":234,"dataGaLocation":479},{"text":236,"config":584},{"href":238,"dataGaName":585,"dataGaLocation":479},"docs",{"text":259,"config":587},{"href":261,"dataGaName":5,"dataGaLocation":479},{"text":254,"config":589},{"href":256,"dataGaName":257,"dataGaLocation":479},{"text":263,"config":591},{"href":265,"dataGaName":266,"dataGaLocation":479},{"text":276,"config":593},{"href":278,"dataGaName":279,"dataGaLocation":479},{"text":268,"config":595},{"href":270,"dataGaName":271,"dataGaLocation":479},{"text":281,"config":597},{"href":283,"dataGaName":284,"dataGaLocation":479},{"text":286,"config":599},{"href":288,"dataGaName":289,"dataGaLocation":479},{"text":291,"config":601},{"href":293,"dataGaName":294,"dataGaLocation":479},{"text":296,"config":603},{"href":298,"dataGaName":299,"dataGaLocation":479},{"title":314,"links":605},[606,608,610,612,614,616,618,622,627,629,631,633],{"text":321,"config":607},{"href":323,"dataGaName":316,"dataGaLocation":479},{"text":326,"config":609},{"href":328,"dataGaName":329,"dataGaLocation":479},{"text":334,"config":611},{"href":336,"dataGaName":337,"dataGaLocation":479},{"text":339,"config":613},{"href":341,"dataGaName":342,"dataGaLocation":479},{"text":344,"config":615},{"href":346,"dataGaName":347,"dataGaLocation":479},{"text":349,"config":617},{"href":351,"dataGaName":352,"dataGaLocation":479},{"text":619,"config":620},"Sustainability",{"href":621,"dataGaName":619,"dataGaLocation":479},"/sustainability/",{"text":623,"config":624},"Diversity, inclusion and belonging (DIB)",{"href":625,"dataGaName":626,"dataGaLocation":479},"/diversity-inclusion-belonging/","Diversity, inclusion and belonging",{"text":354,"config":628},{"href":356,"dataGaName":357,"dataGaLocation":479},{"text":364,"config":630},{"href":366,"dataGaName":367,"dataGaLocation":479},{"text":369,"config":632},{"href":371,"dataGaName":372,"dataGaLocation":479},{"text":634,"config":635},"Modern Slavery Transparency Statement",{"href":636,"dataGaName":637,"dataGaLocation":479},"https://handbook.gitlab.com/handbook/legal/modern-slavery-act-transparency-statement/","modern slavery transparency statement",{"title":639,"links":640},"Contact Us",[641,644,646,648,653,658,663],{"text":642,"config":643},"Contact an expert",{"href":56,"dataGaName":57,"dataGaLocation":479},{"text":383,"config":645},{"href":385,"dataGaName":386,"dataGaLocation":479},{"text":388,"config":647},{"href":390,"dataGaName":391,"dataGaLocation":479},{"text":649,"config":650},"Status",{"href":651,"dataGaName":652,"dataGaLocation":479},"https://status.gitlab.com/","status",{"text":654,"config":655},"Terms of use",{"href":656,"dataGaName":657,"dataGaLocation":479},"/terms/","terms of use",{"text":659,"config":660},"Privacy statement",{"href":661,"dataGaName":662,"dataGaLocation":479},"/privacy/","privacy statement",{"text":664,"config":665},"Cookie preferences",{"dataGaName":666,"dataGaLocation":479,"id":667,"isOneTrustButton":30},"cookie preferences","ot-sdk-btn",{"items":669},[670,672,674],{"text":654,"config":671},{"href":656,"dataGaName":657,"dataGaLocation":479},{"text":659,"config":673},{"href":661,"dataGaName":662,"dataGaLocation":479},{"text":664,"config":675},{"dataGaName":666,"dataGaLocation":479,"id":667,"isOneTrustButton":30},"content:shared:en-us:main-footer.yml","Main Footer","shared/en-us/main-footer.yml","shared/en-us/main-footer",[681],{"_path":682,"_dir":683,"_draft":6,"_partial":6,"_locale":7,"content":684,"config":689,"_id":691,"_type":33,"title":18,"_source":35,"_file":692,"_stem":693,"_extension":38},"/en-us/blog/authors/susie-bitters","authors",{"name":18,"config":685},{"headshot":686,"linkedin":687,"ctfId":688},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1749664195/Blog/Author%20Headshots/susiebittersheadshot.png","https://www.linkedin.com/in/susie-bitters-33268410/","7yiomgeGp9k4a4srjDU1QK",{"template":690},"BlogAuthor","content:en-us:blog:authors:susie-bitters.yml","en-us/blog/authors/susie-bitters.yml","en-us/blog/authors/susie-bitters",{"_path":695,"_dir":41,"_draft":6,"_partial":6,"_locale":7,"header":696,"eyebrow":697,"blurb":698,"button":699,"secondaryButton":703,"_id":705,"_type":33,"title":706,"_source":35,"_file":707,"_stem":708,"_extension":38},"/shared/en-us/next-steps","Start shipping better software faster","50%+ of the Fortune 100 trust GitLab","See what your team can do with the intelligent\n\n\nDevSecOps platform.\n",{"text":49,"config":700},{"href":701,"dataGaName":52,"dataGaLocation":702},"https://gitlab.com/-/trial_registrations/new?glm_content=default-saas-trial&glm_source=about.gitlab.com/","feature",{"text":54,"config":704},{"href":56,"dataGaName":57,"dataGaLocation":702},"content:shared:en-us:next-steps.yml","Next Steps","shared/en-us/next-steps.yml","shared/en-us/next-steps",1754424495583]