How do you show an AI exactly what you mean on a web page?
Phrases such as the second button or that price are ambiguous because pages reorder, hide, and repeat the same words. Point at the control itself. A useful description names what it is, the visible words, where it sits, and the heading above it. Leave out anything you typed into a field.
Why is the second button a poor instruction?
A human looking at a screen shares a moment in time. An assistant often sees a later copy of the same page, a mobile layout, a cookie banner, or a list that has grown by one row. Second then points at a different control. Price is worse still: many amounts sit in the same view, some struck through, some in a cart, some in a footnote.
Role names fail in the same way. Submit, Continue and Next appear more than once. Icons without text look identical to a model that only has a label dump. If you cannot name a unique visible string and a unique nearby heading, you are asking the assistant to guess. Guessing is how the wrong form is posted.
What should a description of a pointed-at element carry?
Carry four facts and stop. First, what the thing is in ordinary language: a primary button, a table cell, a radio option, a link in a nav. Second, the words printed on it, copied exactly, including punctuation. Third, where it sits: left of the total, under the shipping block, in the third column. Fourth, the heading or landmark above it, because that heading is often a unique parent on the page.
Leave out secrets. Anything you typed into a field is not part of the page’s public structure. Passwords, card numbers, one-time codes and private notes must not travel with a pointer. The assistant needs the control, not the payload you already entered. If the field is empty, say it is empty rather than pasting a value.
- Control type
- A short name for the widget: button, link, checkbox, cell, tab, or heading, not a guess at the framework.
- Visible text
- The exact string on the control, including case and punctuation, or none if it is an icon-only control.
- Place
- A neighbour the eye can check: under this heading, next to this label, in this column.
- Landmark
- The nearest heading, tab name or section title that would still identify the block if the rest of the page shuffled.
How do you point when the control has no words?
Icon-only buttons need a neighbour with words. Name the labelled control beside them, then the direction: the trash icon to the right of Invoice 1842. If the only cue is colour or position in a toolbar, say the toolbar’s accessible name if the page shows one, or the last visible heading above that toolbar.
Repeated rows need a key from the row, not an index. Prefer the order reference, the email address shown, or the product title in that row. Indexes break when a filter is on, when a row is inserted, or when a sticky header eats a count. If two rows share the same title, add the second unique cell in that row.
| Include | Omit | Why |
|---|---|---|
| Control type and exact visible words | Anything typed into an input | The page structure is public; field values are not |
| Nearest heading or section title | Pixel coordinates and screenshots as the only cue | Headings survive layout change; pixels do not |
| A unique cell in the same row | First, second, last, or that one | Order changes with sort, filter and banners |
When is a simpler note better than a full pointer?
If the page has one obvious primary action and you are sitting in front of it, a short sentence is enough: press Save under Account. Do not build a four-part description for a unique heading. Extra structure helps when the view is dense, when the same label repeats, or when someone else will replay the instruction later.
A screenshot alone is a weak pointer. It freezes one layout and hides live text that a reader can copy. Use a picture only to show a spatial relation you cannot name, and still write the heading and the visible words. GROX is a single conversational surface that can act in tools you already use; it still needs those four facts when the page is crowded. If you only need a reminder for yourself, a paper note beside the monitor is the better tool.
Common questions
Why does that price confuse an assistant on a web page?
Price is a common word and a common pattern. The same view may show a list price, a sale price, a unit price, tax and a running total. Without the heading above the figure and the neighbouring label, the assistant cannot know which amount you meant. Point at the cell and name the heading, not the idea of a price.
What four facts should a pointed-at element include?
Name the control type, copy the visible words exactly, say where it sits relative to a neighbour, and name the heading or landmark above it. Those four facts usually survive a layout change. Do not send field values you typed. If the control is an icon, name a labelled neighbour instead of inventing a caption.
Should I include text I typed into a form field?
No. Typed values are not part of the public structure of the page and may be secrets. Describe the empty or filled field by its label and heading, then give the action: clear it, leave it, or submit. If a value must be used, supply it in a separate instruction after the pointer, not as if it were printed on the control.
When should I skip a detailed pointer and just name the button?
Skip the long form when the page has a single matching label under a unique heading and you are looking at that page now. A short line is then clearer. Use the full description when labels repeat, when the layout may change, or when someone else will run the same step later without seeing your screen.
For how a persistent agent routes chat into real tools, see Home and the Help Centre.