{"schema_version":1,"research_id":"eoa_inverse_innovation_exp05_external_evaluation_20260803","source_assessment_id":"negative_space_design__rhetoric:P4:v0","cell_id":"negative_space_design__rhetoric","search_queries":["annotation first reading rhetorical analysis students annotations influence attention study","pretraining signaling principle annotations attention learning primary research highlights text study","social annotation education rhetorical reading platform annotations hidden first pass","WCAG supplemental content user control annotations accessibility guidance","site:perusall.com instructor hide annotations until student posts original response","site:web.hypothes.is \"Paced Social Annotation\" launch date","site:w3.org/WAI/WCAG22 Understanding status messages user controls timing content reveal","site:ada.gov web mobile app accessibility rule public universities WCAG 2.1 AA 2024","site:bls.gov/oes/current national occupational employment wage estimates instructional coordinators web developers 2025","site:bls.gov/ooh instructional coordinators median pay web developers digital designers median pay 2025","\"How the Quality of Annotation Influences Academic Reading\" full text 47 students 2025","annotation salience eye tracking students attention primary research text highlights annotations","site:bls.gov/oes/2025/may/oes151254.htm Web Developers May 2025 mean wage","site:bls.gov/oes/2025/may/oes259031.htm Instructional Coordinators May 2025 wage","site:bls.gov/oes/current/oes151254.htm web developers annual mean wage May 2025","site:bls.gov/oes/current/oes259031.htm instructional coordinators annual mean wage May 2025"],"sources":[{"source_id":"S1","title":"Marginal Pedagogy: How Annotated Texts Affect a Writing-from-Sources Task","publisher":"Written Communication / ERIC, U.S. Department of Education","url":"https://eric.ed.gov/?id=EJ779584","source_class":"PRIMARY_RESEARCH","publication_date":"2002","accessed_at":"2026-08-03","claims_supported":["In a study of more than 120 students, evaluative annotations changed how persuasive identical passages were perceived to be.","Annotations affected the rhetorical task and sometimes caused inclusion of irrelevant information, although they also supported some advanced engagement.","Visible annotation can influence interpretation rather than merely add neutral support."]},{"source_id":"S2","title":"How the Quality of Annotation Influences Academic Reading: An Eye-Tracking Study","publisher":"Journal of Computer Assisted Learning / ERIC, U.S. Department of Education","url":"https://eric.ed.gov/?id=EJ1474661","source_class":"PRIMARY_RESEARCH","publication_date":"2025-06","accessed_at":"2026-08-03","claims_supported":["A randomized laboratory study with 47 college students found that annotations of different cognitive quality produced different reading patterns.","Constructive annotations attracted more attention than active annotations.","Annotation content can direct visual attention and processing, while potentially supporting comprehension."]},{"source_id":"S3","title":"Early Release Info — Paced Social Annotation","publisher":"Hypothesis","url":"https://web.hypothes.is/help/early-release-info-paced-social-annotation/","source_class":"OFFICIAL_PRODUCT_DOCUMENTATION","publication_date":"n.d.","accessed_at":"2026-08-03","claims_supported":["Hypothesis already offers a Paced Social Annotation mode intended to obtain more original student comments on a first pass and permit later comparison with others' comments.","The product implements hidden and revealed annotation states, a visible state indicator, an instructor-controlled checkpoint, persistent document display, and LMS workflows.","The documentation identifies instructors as adopters and explicitly articulates the need for original first-pass comments, rereading, comparison, and staged scaffolding.","The current documented design may leave instructor annotations visible and gives the instructor, rather than each reader, the reveal control."]},{"source_id":"S4","title":"Making Reading Visible: Social Annotation with Lacuna in the Humanities Classroom","publisher":"Stanford University","url":"https://tec.fsi.stanford.edu/publication/making-reading-visible-social-annotation-lacuna-humanities-classroom","source_class":"PRIMARY_RESEARCH","publication_date":"2016-06-16","accessed_at":"2026-08-03","claims_supported":["Lacuna was designed and deployed for critical reading and dialogue in humanities courses.","A comparative-literature case study found that source-linked student annotations gave instructors insight into student perspectives.","Social annotation introduced additional responsibilities, workflows, and self-reflection demands."]},{"source_id":"S5","title":"Spotlights: Designs for Directing Learners’ Attention in a Large-Scale Social Annotation Platform","publisher":"Proceedings of the ACM on Human-Computer Interaction / MIT","url":"https://people.csail.mit.edu/farnazj/pdfs/spotlights_CSCW2022.pdf","source_class":"PRIMARY_RESEARCH","publication_date":"2022-11","accessed_at":"2026-08-03","claims_supported":["Social-annotation systems commonly place highlights and comments beside or within readings and use visual cues to make annotations salient.","Hundreds of instructors had used the NB platform, demonstrating technical and workflow scalability of anchored annotation.","Randomized spotlight treatments changed comment-click likelihood, directly demonstrating that annotation presentation can direct attention.","Large comment volumes can overwhelm students and cause disengagement."]},{"source_id":"S6","title":"Web Content Accessibility Guidelines (WCAG) 2.2","publisher":"World Wide Web Consortium","url":"https://www.w3.org/TR/WCAG22/","source_class":"STANDARD","publication_date":"2023-10-05","accessed_at":"2026-08-03","claims_supported":["Equivalent text alternatives, perceivable content, understandable operation, keyboard access, adequate timing, and programmatically determinable control state are relevant implementation requirements.","A reveal control, annotation-state indicator, scratchpad, source alignment, and timed transition must remain usable with assistive technology.","Accessibility content cannot safely be treated as optional interpretive clutter."]},{"source_id":"S7","title":"Accessibility of Web Content and Mobile Apps Provided by State and Local Government Entities: A Small Entity Compliance Guide","publisher":"U.S. Department of Justice, Civil Rights Division","url":"https://www.ada.gov/resources/small-entity-compliance-guide/","source_class":"GOVERNMENT_OR_REGULATOR","publication_date":"2026-04-20","accessed_at":"2026-08-03","claims_supported":["Title II applies to public schools, community colleges, and public universities, including vendor-provided web content under contractual or licensing arrangements.","WCAG 2.1 Level AA is the applicable technical standard under the rule, subject to the updated compliance dates stated in the guidance.","Public entities retain effective-communication, reasonable-modification, and equal-participation obligations even before specific rule deadlines.","Institutions remain responsible for accessibility when using an external learning-platform vendor."]},{"source_id":"S8","title":"Occupational Employment and Wages — May 2025","publisher":"U.S. Bureau of Labor Statistics","url":"https://www.bls.gov/news.release/archives/ocwage_05152026.pdf","source_class":"OFFICIAL_ORGANIZATION_DATA","publication_date":"2026-05-15","accessed_at":"2026-08-03","claims_supported":["May 2025 national mean wages were $33.54 per hour across occupations and $47.49 per hour for web developers.","The official wage data provide a defensible labor-rate anchor for 2026 resource-equivalent estimates.","The estimates are employer-survey data and do not constitute product-license or contractor quotations."]}],"problem_evidence":{"support":"MODERATE","rationale":"The problem is visible and consequential: controlled studies show that annotations can alter passage persuasiveness, attract attention, and change reading behavior, while large annotation volumes can become overwhelming. However, no opened source directly measures the prevalence or magnitude of apparatus-driven cue-following specifically in rhetorical-criticism first encounters, and annotations can also improve comprehension. The problem therefore exists, but its frequency and net harm in the proposed setting remain uncertain.","source_ids":["S1","S2","S5"]},"stakeholder_evidence":{"support":"STRONG","rationale":"Hypothesis identifies instructors as adopters and explicitly markets a staged hidden/revealed workflow to obtain original first-pass comments and enable later comparison. Stanford's humanities deployment independently shows instructors using source-linked annotation to understand student perspectives. Instructors, platform owners, and public-university accessibility/procurement authorities are identifiable authorizers, although no named institution has expressed demand for every guardrail in this candidate's bundle.","source_ids":["S3","S4","S7"]},"prior_art":{"proximity":"ESTABLISHED_PRACTICE","closest_analogues":[{"name":"Hypothesis Paced Social Annotation","similarity":"Direct product-level analogue: hides student annotations during an initial pass, captures original source-linked student comments, displays hidden/revealed status, and later reveals annotations for comparison and rereading.","remaining_difference":"The candidate additionally proposes hiding instructor/model interpretations, retaining a neutral rhetorical-observation schema, allowing each reader immediate reveal or skip, preserving specified contextual layers, and aligning pre-reveal observations with restored scholarly annotations. These are configuration and governance differences around an already implemented core sequence.","source_ids":["S3"]},{"name":"Lacuna social annotation in humanities seminars","similarity":"Existing humanities workflow makes student reading and source-linked interpretations visible to instructors and connects reading activity to later discussion.","remaining_difference":"The documented Lacuna case emphasizes shared visible annotation and dialogue rather than a protected, annotation-free first state followed by controlled restoration.","source_ids":["S4"]},{"name":"NB/Spotlights social-annotation designs","similarity":"Implements anchored comments, margins, highlights, attention cues, logging, and large-course workflows; experiments show that presentation changes what annotations students inspect.","remaining_difference":"Spotlights amplifies selected annotations rather than withholding the interpretive layer to preserve an unaided first record.","source_ids":["S5"]}],"distinctive_claim_remaining":"Relative to Hypothesis Paced Social Annotation, a first state that hides peer, instructor, and model interpretations while preserving predefined contextual and accessibility layers, gives each reader an immediate reveal-or-skip option, and captures neutral source-linked rhetorical observations will produce less reuse of supplied annotation language without reducing passage grounding, orientation, accessibility, or post-reveal comprehension. This is narrow and falsifiable, but no source establishes its advantage.","confidence":"HIGH"},"implementation_evidence":{"support":"STRONG","rationale":"The core state machine, anchored comments, LMS delivery, checkpoints, state indicators, and later reveal already exist in a first-party product. Research systems demonstrate stable annotation anchoring and attention logging at substantial classroom scale. A bounded pilot is within instructor or platform-owner authority for controlled materials. Feasibility conditions include permission-cleared text; explicit consent and data minimization; institution-approved storage; no grading or sensitive-belief inference; reader-controlled reveal; no fixed time disadvantage; preservation of provenance, warnings, translations, captions, descriptions, keyboard navigation, and assistive controls; and accessibility testing of hidden/revealed state changes. FERPA, local research-review requirements, responsive-layout anchoring, cross-translation alignment, and product licensing were not resolved by the eight sources.","source_ids":["S3","S4","S5","S6","S7"]},"scores":{"meaningful_impact":{"score":3,"rationale":"Annotations demonstrably influence attention and judgments, so separating unaided noticing from prompted recognition could improve instructional interpretation. Rhetoric-specific prevalence and realized educational benefit are unmeasured, and annotations may also help comprehension.","source_ids":["S1","S2","S5"]},"stakeholder_pull":{"score":4,"rationale":"A commercial educational-annotation provider explicitly describes instructor demand for original first-pass comments and later comparison; humanities instructors have adopted related workflows.","source_ids":["S3","S4"]},"incremental_advantage":{"score":2,"rationale":"The all-interpretive-layer hiding rule, context taxonomy, individual reveal, and neutral rhetorical scratchpad are testable refinements, but no evidence shows that they outperform the existing paced product or ordinary teaching configurations.","source_ids":["S3"]},"distinctiveness_plausibility":{"score":1,"rationale":"The defining staged first-pass/capture/reveal sequence is already documented as a live product feature. The remaining distinction is principally configuration, governance, and measurement detail.","source_ids":["S3"]},"technical_implementability":{"score":5,"rationale":"The major technical elements already operate in Hypothesis and related social-annotation systems; remaining work is accessible configuration, note-schema design, anchoring, and institutional integration.","source_ids":["S3","S5","S6"]},"adoption_authority_feasibility":{"score":4,"rationale":"Instructors and platform owners can authorize a non-graded pilot using controlled material, while institutional IT, accessibility, privacy, procurement, and possibly research-review authorities govern deployment and data collection.","source_ids":["S3","S4","S7"]},"evidence_readiness":{"score":4,"rationale":"The problem mechanisms, existing systems, observable outcomes, comparators, and implementation standards are well documented. The candidate's remaining incremental effect requires participant-level testing and is not web-answerable.","source_ids":["S1","S2","S3","S5","S6"]},"safety_net_benefit":{"score":3,"rationale":"Immediate reveal, preserved context, untimed completion, and accessible controls can limit confusion or exclusion, but those protections have not been evaluated as a bundle and public institutions retain case-specific accessibility duties.","source_ids":["S6","S7"]},"scalability":{"score":5,"rationale":"LMS-integrated and large-class social-annotation systems already demonstrate scalable delivery, anchored comments, state management, and instructor workflows.","source_ids":["S3","S5"]}},"score_confidence":"HIGH","costs":{"first_evidence":{"band_2026_usd":"UNDER_10K","scope":"Documentary fit-gap audit plus a 12-to-16-reader, non-graded feasibility study using two permission-cleared excerpts, existing software, source-linked coding, modest incentives, and an accessibility check.","confidence":"MODERATE","assumptions":["Approximately 60 to 120 combined instructor, coordinator, coder, and technical hours.","Existing institutional LMS and annotation access are available without a new enterprise contract.","Participant incentives and accessibility accommodations remain modest.","BLS mean hourly rates of roughly $34 across occupations and $47 for web developers are used only as resource-equivalent anchors."],"source_ids":["S3","S8"]},"initial_deployment_startup":{"band_2026_usd":"10K_TO_50K","scope":"Configure one department's LMS workflow, observation template, data retention, permission handling, accessibility verification, staff guidance, and rollback process using an existing annotation platform.","confidence":"LOW","assumptions":["Approximately 200 to 500 cross-functional hours.","No custom annotation engine is built.","Vendor licensing, security review, and procurement fees are not publicly quoted and could move the estimate.","Scope is one department, not institution-wide deployment."],"source_ids":["S3","S6","S7","S8"]},"operational_launch":{"band_2026_usd":"10K_TO_50K","scope":"Launch across several courses in one department, including instructor training, accessible material preparation, technical support, privacy review, monitoring, and evaluation.","confidence":"LOW","assumptions":["Approximately 300 to 800 staff hours distributed across curriculum, IT, accessibility, privacy, and support roles.","Existing LMS integration is reusable across courses.","No bespoke mobile application or large-scale migration is required.","Commercial license costs remain an unverified component."],"source_ids":["S3","S7","S8"]},"annual_recurring":{"band_2026_usd":"10K_TO_50K","scope":"Annual platform access allocation, course setup, annotation maintenance, accessibility regression checks, instructor support, retention administration, and outcome review for one department.","confidence":"LOW","assumptions":["Approximately 200 to 500 recurring staff hours plus an unknown share of platform licensing.","Source documents and annotation alignments require periodic maintenance.","Course count and procurement terms stay within departmental scale.","Institution-wide licensing or custom development would require a new estimate."],"source_ids":["S3","S6","S7","S8"]}},"verified_pipeline_gates":{"externally_supported_problem":{"status":"YES","reason":"Primary studies show that annotations can alter rhetorical judgments and reading attention, establishing the mechanism even though rhetoric-course prevalence is unknown.","source_ids":["S1","S2","S5"]},"externally_credible_adopter_or_authorizer":{"status":"YES","reason":"Hypothesis explicitly serves instructors seeking original first-pass comments, and humanities instructors have deployed related social-reading systems; instructors and platform owners are identifiable authorizers.","source_ids":["S3","S4"]},"distinct_testable_incremental_claim":{"status":"YES","reason":"The narrow remaining comparison against Hypothesis is testable through supplied-language reuse, passage grounding, orientation, accessibility, reveal behavior, and post-reveal comprehension, although its advantage is unsupported.","source_ids":["S3"]},"bounded_next_evidence_step":{"status":"YES","reason":"A time-boxed documentary fit-gap audit can compare the candidate requirement matrix with Hypothesis Paced Social Annotation, a conventional visibility toggle, and a clean-copy workflow, with an explicit collision falsifier.","source_ids":["S3","S4","S5"]},"no_unresolved_safety_or_authority_stop":{"status":"YES","reason":"A non-graded, permission-cleared, non-sensitive, reader-controlled pilot can be authorized with consent, immediate reveal, accessible controls, and institutional privacy/research review. Unresolved local compliance details constrain implementation but are not an inherent stop.","source_ids":["S6","S7"]},"credible_cost_scope_and_range":{"status":"YES","reason":"The estimates are explicitly bounded to one department using an existing platform and are anchored to current official labor data. They are resource-equivalent ranges, not vendor quotations; licensing uncertainty is disclosed.","source_ids":["S3","S8"]}},"next_evidence_step":"Before any new build, run a four-hour documentary fit-gap audit. Create a requirement matrix covering hidden peer/instructor/model layers, preserved context, individual reveal and skip, neutral source-linked capture, stable post-reveal alignment, export, accessibility, timing, privacy, and rollback. Score three comparators from their public documentation: Hypothesis Paced Social Annotation, an ordinary annotation-visibility toggle, and a clean-copy-then-annotated-copy workflow. The proposal's distinctiveness is falsified if Hypothesis or a configuration of it already satisfies every load-bearing requirement; its adoption fit is falsified if the institution cannot preserve accessibility or reader-controlled recovery. Only if a material gap survives should a separate consenting field study compare the candidate configuration with Hypothesis's standard paced mode; that later effect evidence would require live testing.","blocking_evidence":["No rhetoric-specific prevalence estimate shows how often simultaneous annotations prevent interpretable independent first observations.","No field evidence shows that hiding instructor and model annotations, adding individual reveal, or using the neutral scratchpad improves outcomes over Hypothesis Paced Social Annotation.","Product licensing, procurement, privacy/FERPA treatment, local research-review requirements, and exact institutional labor costs remain unverified.","Accessibility of the complete hidden/revealed interaction, source anchoring across responsive layouts and translations, and effects on readers needing scaffolding require user testing.","World novelty, patentability, freedom to operate, market size, and realized impact were not measured."],"research_disposition":"KNOWN_PRACTICE_DIFFUSION","world_novelty_boundary":"World novelty is unmeasured. This bounded eight-source search establishes a direct live product analogue and several adjacent research systems; it is not an exhaustive prior-art, patentability, freedom-to-operate, market-size, or realized-impact assessment.","arm":"COMPLETE_PROPOSAL_PORTFOLIO","candidate_version":0,"controller_recommendation":{"action":"STOP_DIFFUSION","repairable":false,"material_progress_observed":false,"progress_targets":["Reclassify the staged first-pass/capture/reveal sequence as known-practice diffusion rather than a new intervention.","Use a documented fit-gap review of Hypothesis Paced Social Annotation before authorizing any custom build.","If adopting, preserve reader-controlled recovery, accessibility supports, necessary context, privacy review, and non-punitive use of first-pass notes.","Treat any claimed advantage from hiding instructor annotations or adding the rhetorical scratchpad as a separate empirical research question, not as an established benefit."],"reason":"The core proposal substantially matches Hypothesis's documented Paced Social Annotation product: an initial hidden-annotation phase intended to elicit original comments, followed by reveal, rereading, and comparison. Remaining differences are narrow configuration and governance refinements without evidence of incremental benefit. The appropriate path is adoption or diffusion of known practice, not continuation through an innovation pipeline."},"proposal_index":4}