pandas – python shift function does not work as expected

I have a data frame for the page events with the following structure:

  • sessionId – unique session id
  • event_date – date and time where the event occurred
  • pageTitle – event name (page to which users navigated)

In order to see, how users navigate in app and build a Sankey chart for it, I would like to create a new column “next event” which would be recording the next event for the session using the shift function, but the code below returns NAN values or mixing everything and display wrong next event (I see it from event_date).

df('event_date') = pd.to_datetime(df('event_date'), unit='s')
df.sort_values(('sessionId', 'pageTitle', 'event_date'),
                 ascending=(True, False, True), inplace=True)


grouped = df.groupby('sessionId')

def get_next_event(x): return x('pageTitle').shift(-1)

df("next_event") = grouped.apply(
    lambda x: get_next_event(x)).reset_index(0, drop=True)

display(df.query('sessionId =="1"').sort_values('event_date'))
display(df.query('sessionId =="2"').sort_values('event_date'))

results example

sessionId event_date pageTitle next_event
1 2021-05-26 19:23:45.820 Search Search
1 2021-05-26 19:23:50.074 Statement Statement
1 2021-05-26 19:30:06.086 Search Search
1 2021-05-26 19:30:09.995 Statement Statement
1 2021-05-26 19:47:24.058 Statement Statement
1 2021-05-26 19:57:31.661 Aging Report NAN
1 2021-05-26 19:57:40.672 Search Aging Report
1 2021-05-26 19:57:44.160 Statement Search

2nd

sessionId event_date pageTitle next_event
2 2021-07-20 15:43:35.941 Aging Report NaN
2 2021-07-20 15:44:52.739 Search Search
2 2021-07-20 15:44:56.173 Statement Statement
2 2021-07-20 16:23:02.761 Statement Statement

Please help to understand what I am doing wrong