Outline

• About the project
• Technical features
• Special challenges
• Solutions
• Docker tutorial
• Sitediff demo

About the project

• McGill University's Course Calendar (aka Catalogue)
• Programs, Courses, and University Regulations
• Legal documents, course schedules, metadata, cross-referencing
• Search-driven UI

Technical features (UI)

• search-driven navigation
  • tabs (heirarchical facets)
  • custom facets
  • indexing embedded content (via node refs)
  • search this section (menu item)
  • performance
• UI: all data lives in logical tree hierarchy
  • think 10k primary link items LOGICALLY, but really the tree is defined by biz logic
  • menus (custom code, core, menu_block)
  • breadcrumbs, context, flattening

Technical features (Data)

• including revision, site has 70k nodes, imported from banner+documentum
• 15 content types, 170 field instances
• a lot of cross-linking via node references
• kind-of i18n (hacks for continuing studies)
• web services
• input filters (auto-detection of course names in any HTML content)

Extra requirements

• Custom modules
  • Legacy => really really custom (4+ years of code drift)
  • Extended apachesolr 6.x-2.x-dev
• Contrib Modules
  • Had to deduce which version of contrib modules were enabled from system table
  • Many contrib modules were enabled but unused
• Concern about data and configuration integrity
• Test driven development
• Deliverable = upgrade script, not code + db dump

Incidental challenges

• Defeaturization
• Deploying a dev site (not the same as prod?)
  • missing content types, modules, blocks
• content migrate running time
  • prune the database (10% of the nodes, focused on 1 faculty, try to keep consistency)
  • contentmigratetweaks https://github.com/dergachev/contentmigratetweaks
• git branch hecticness
• i18nfield allowedvalues translation bug
• misc migration bugs (entityreference, nodeblock)

Technical solutions

• Refactoring first for sanity
• Docker for build process
• Unit tests for the new code we write; phpunit for filters and menus
• Sitediff for correctness
• Content migrate tweaks for speed
• Search API FTW!

Unit testing

• phpunit tests per module (works with D7 "OK")
  • bootstrap drupal => can't mock global functions, need process isolation
  • instead, use fixtures and mocks in your tests w/o drupal process isolation, can't mock global functions
• autoloading: explored composer, manually, PSR0
• fixtures are awesome: menus, nodes

Scripted upgrade process

• D6 deploy script
• d6 refactor adjustments
• d6 prepare (defeaturize, turn off non-core modules, change theme to bartik, pm-uninstall several modules)
• core updb
• enable modules, run contrib updb
• content migrate
• menu_adjustments.php
• d7_adjustments.sh
• d7_adjustments_solr.sh
• all glued together by Makefile and multiple Dockerfile-s

Docker for build process

• instantly spin-up of container (lightweight VM) with EXACT dev environment
• easily script build of container
• smart caching FTW!
• sipmlicity (vs chef): just know bash
• copy on write - instant snapshots
• https://github.com/amirkdv/drupal-docker
• Makefile

Docker tutorial / demo

• https://github.com/dergachev/drupal-docker-marriage/

SiteDiff

• bookmarklet
• show pretty diffs of HTML contents of whole sites
• sanitization - easy to write! easy to maintain!

SiteDiff Input

    paths:
     - /
     - /user

    selector: 'body'

    sanitization:
    - title: 'remove form build id'
      pattern:    '<input type="hidden" name="form_build_id" value="form-[a-zA-Z0-9_-]+" *\/?>'
      substitute: '<input type="hidden" name="form_build_id" value="__form_build_id__">'

    before:
      dom_transform:
      - type: remove
        selector: '#something'

SiteDiff output

Content migrate tweaks

• https://github.com/dergachev/content_migrate_tweaks
• content_migrate submodule of CCK is sloooow
• one field at a time, one node at a time, one value at a time
• stopped it after about 10 hours
• query optimization
• unit and integration testing

Any questions about...?

• D7 Upgrades
• Content migrate
• Unit testing
• Docker
• Sitediff