[Crash-utility] Unable to switch stack frames while using crash

[Shashidhara Shamaiah]

Hi,



I was investigating a 64 bit linux kernel dump . I have following doubts
regarding usage of crash.



1) I wanted to access the intermediate kernel stack frames. To know the
status of the frame and the point of failure.

When I tried to access a stack frame I get an error message "crash:
prohibited gdb command: frame". Can you please let me know if there is
any other way of accessing the kernel stack frames using
crash.



2) When I run bt in crash, I get a stack trace. Another person from a
different team reported a slightly different stack trace to mine. Below
are the stack traces. The register contents are quite different between
the two



My stack trace

PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"

#0 [ffff88031ce759d0] machine_kexec at ffffffff81024486

#1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230

#2 [ffff88031ce75b20] oops_end at ffffffff8100fa38

#3 [ffff88031ce75b50] no_context at ffffffff8102d801

#4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9

#5 [ffff88031ce75c70] bad_area at ffffffff8102da41

#6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19

#7 [ffff88031ce75cf0] page_fault at ffffffff812d7425

#8 [ffff88031ce75d78] n_tty_read at ffffffff811f03b3

#9 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e

#10 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8

#11 [ffff88031ce75f40] sys_read at ffffffff810ebe48

#12 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2

RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212

RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000

RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000

RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000

R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000

R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0

ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b





Reported stack trace

PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"

#0 [ffff88031ce759d0] machine_kexec at ffffffff81024486

#1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230

#2 [ffff88031ce75ad8] n_tty_read at ffffffff811f03b3

#3 [ffff88031ce75b20] oops_end at ffffffff8100fa38

#4 [ffff88031ce75b50] no_context at ffffffff8102d801

#5 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9

#6 [ffff88031ce75c20] native_sched_clock at ffffffff810120aa

#7 [ffff88031ce75c70] bad_area at ffffffff8102da41

#8 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19

#9 [ffff88031ce75cf0] page_fault at ffffffff812d7425

[exception RIP: n_tty_read+1420]

RIP: ffffffff811f03b3 RSP: ffff88031ce75da8 RFLAGS: 00010246

RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044

RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c

RBP: ffff88031ce75eb8 R8: 0000000000000000 R9: 0000000000000000

R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044

R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff

ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #10
[ffff88031ce75ec0] tty_read at ffffffff811ebf7e

#11 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8

#12 [ffff88031ce75f40] sys_read at ffffffff810ebe48

#13 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2





3) I want to retrieve the address of a data structure in the current
context. How can it be done? I tried using struct command, but it did
not help



4) When I run the command readelf -a vmcore, I get an error message
"readelf: Error: Not an ELF file - it has the wrong magic bytes at the
start."





Please help regarding the above queries.



Thanks and Regards

Shashidhara


Information transmitted by this e-mail is proprietary to MphasiS, its associated companies and/ or its customers and is intended
for use only by the individual or entity to which it is addressed, and may contain information that is privileged, confidential or
exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded
to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly
prohibited. In such cases, please notify us immediately at ***@mphasis.com and delete this mail from your records.

[Flavio Leitner]

Post by Shashidhara Shamaiah
Hi,
I was investigating a 64 bit linux kernel dump . I have following doubts
regarding usage of crash.
1) I wanted to access the intermediate kernel stack frames. To know the
status of the frame and the point of failure.
prohibited gdb command: frame". Can you please let me know if there is
any other way of accessing the kernel stack frames using
crash.

Try 'bt -f'
Try also 'help bt' to discover other nice parameters as well.

Post by Shashidhara Shamaiah
2) When I run bt in crash, I get a stack trace. Another person from a
different team reported a slightly different stack trace to mine. Below
are the stack traces. The register contents are quite different between
the two
My stack trace
PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"
#0 [ffff88031ce759d0] machine_kexec at ffffffff81024486
#1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230
#2 [ffff88031ce75b20] oops_end at ffffffff8100fa38
#3 [ffff88031ce75b50] no_context at ffffffff8102d801
#4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9
#5 [ffff88031ce75c70] bad_area at ffffffff8102da41
#6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19
#7 [ffff88031ce75cf0] page_fault at ffffffff812d7425
#8 [ffff88031ce75d78] n_tty_read at ffffffff811f03b3
#9 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e
#10 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8
#11 [ffff88031ce75f40] sys_read at ffffffff810ebe48
#12 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2
RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212
RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000
RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000
RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
Reported stack trace
PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"
#0 [ffff88031ce759d0] machine_kexec at ffffffff81024486
#1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230
#2 [ffff88031ce75ad8] n_tty_read at ffffffff811f03b3
#3 [ffff88031ce75b20] oops_end at ffffffff8100fa38
#4 [ffff88031ce75b50] no_context at ffffffff8102d801
#5 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9
#6 [ffff88031ce75c20] native_sched_clock at ffffffff810120aa
#7 [ffff88031ce75c70] bad_area at ffffffff8102da41
#8 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19
#9 [ffff88031ce75cf0] page_fault at ffffffff812d7425
[exception RIP: n_tty_read+1420]
RIP: ffffffff811f03b3 RSP: ffff88031ce75da8 RFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044
RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c
RBP: ffff88031ce75eb8 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044
R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #10
[ffff88031ce75ec0] tty_read at ffffffff811ebf7e
#11 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8
#12 [ffff88031ce75f40] sys_read at ffffffff810ebe48
#13 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2
3) I want to retrieve the address of a data structure in the current
context. How can it be done? I tried using struct command, but it did
not help

You need to use bt -f and find what pointer you are looking for and
then: struct <struct name> pointer
most of times just doing: <struct name> pointer
works out

fbl

Post by Shashidhara Shamaiah
4) When I run the command readelf -a vmcore, I get an error message
"readelf: Error: Not an ELF file - it has the wrong magic bytes at the
start."
Please help regarding the above queries.
Thanks and Regards
Shashidhara
Information transmitted by this e-mail is proprietary to MphasiS, its associated companies and/ or its customers and is intended
for use only by the individual or entity to which it is addressed, and may contain information that is privileged, confidential or
exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded
to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly
--
Crash-utility mailing list
https://www.redhat.com/mailman/listinfo/crash-utility

[Dave Anderson]

----- Original Message -----

Post by Shashidhara Shamaiah
Hi,
I was investigating a 64 bit linux kernel dump . I have following
doubts regarding usage of crash.
1) I wanted to access the intermediate kernel stack frames. To know
the status of the frame and the point of failure.
prohibited gdb command: frame ”. Can you please let me know if there
is any other way of accessing the kernel stack frames using crash.

Right -- the embedded gdb doesn't know anything about the core file
(or live system) that you're running on. It's invoked as "gdb vmlinux",
and doesn't know anything about any "frames".

As Flavio mentioned, you can see the stack data of each frame with
"bt -f", or better yet, "bt -F" which may illuminate what the data
may be, because it shows symbolic translations or slab cache names
instead of raw values where appropriate.

Post by Shashidhara Shamaiah
2) When I run bt in crash, I get a stack trace. Another person from a
different team reported a slightly different stack trace to mine.
Below are the stack traces. The register contents are quite different
between the two
My stack trace
PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"
#0 [ffff88031ce759d0] machine_kexec at ffffffff81024486
#1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230
#2 [ffff88031ce75b20] oops_end at ffffffff8100fa38
#3 [ffff88031ce75b50] no_context at ffffffff8102d801
#4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9
#5 [ffff88031ce75c70] bad_area at ffffffff8102da41
#6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19
#7 [ffff88031ce75cf0] page_fault at ffffffff812d7425
#8 [ffff88031ce75d78] n_tty_read at ffffffff811f03b3
#9 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e
#10 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8
#11 [ffff88031ce75f40] sys_read at ffffffff810ebe48
#12 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2
RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212
RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000
RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000
RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
Reported stack trace
PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"
#0 [ffff88031ce759d0] machine_kexec at ffffffff81024486
#1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230
#2 [ffff88031ce75ad8] n_tty_read at ffffffff811f03b3
#3 [ffff88031ce75b20] oops_end at ffffffff8100fa38
#4 [ffff88031ce75b50] no_context at ffffffff8102d801
#5 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9
#6 [ffff88031ce75c20] native_sched_clock at ffffffff810120aa
#7 [ffff88031ce75c70] bad_area at ffffffff8102da41
#8 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19
#9 [ffff88031ce75cf0] page_fault at ffffffff812d7425
[exception RIP: n_tty_read+1420]
RIP: ffffffff811f03b3 RSP: ffff88031ce75da8 RFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044
RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c
RBP: ffff88031ce75eb8 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044
R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #10 [ffff88031ce75ec0]
#10 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e
#11 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8
#12 [ffff88031ce75f40] sys_read at ffffffff810ebe48
#13 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2

The first backtrace is different because you are apparently using an
older version of the crash utility, because it is not showing the
page fault exception frame like the "reported" version.

Post by Shashidhara Shamaiah
3) I want to retrieve the address of a data structure in the current
context. How can it be done? I tried using struct command, but it did
not help

The struct command needs the correct virtual address of the structure
you're trying to view. So I presume you're asking how to find the address
of the data structure? If that's true, you're going to have to be a lot
more specific.

Post by Shashidhara Shamaiah
4) When I run the command readelf -a vmcore, I get an error message
”readelf: Error: Not an ELF file - it has the wrong magic bytes at the
start.”

I presume that the dumpfile is a compressed kdump dumpfile generated
by makedumpfile, which takes the original /proc/vmcore ELF dumpfile
and creates its own unique dumpfile format.

Dave

[Shashidhara Shamaiah]

Hi Dave,

Thanks for the help, I have further input regarding query 3 . Please help.

Post by Shashidhara Shamaiah
3) I want to retrieve the address of a data structure in the current
context. How can it be done? I tried using struct command, but it did
not help

The struct command needs the correct virtual address of the structure
you're trying to view. So I presume you're asking how to find the address
of the data structure? If that's true, you're going to have to be a lot
more specific.

Post by Shashidhara Shamaiah

I need to find out the virtual address of the structure tty of type struct tty_struct, which is passed as an argument to the function n_read_tty. Below is the corresponding stack trace.
PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"
#0 [ffff88031ce759d0] machine_kexec at ffffffff81024486
#1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230
#2 [ffff88031ce75b20] oops_end at ffffffff8100fa38
#3 [ffff88031ce75b50] no_context at ffffffff8102d801
#4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9
#5 [ffff88031ce75c70] bad_area at ffffffff8102da41
#6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19
#7 [ffff88031ce75cf0] page_fault at ffffffff812d7425
#8 [ffff88031ce75d78] n_tty_read at ffffffff811f03b3
#9 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e
#10 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8
#11 [ffff88031ce75f40] sys_read at ffffffff810ebe48
#12 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2
RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212
RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000
RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000
RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
I have another query I tried to convert the vmcore file to ELF format using "makedumpfile -E -d 31 -x vmlinux_temp vmcore dumpfile" . For which I got an error message " '-E' option is disable, because vmcore is kdump compressed format.

makedumpfile Failed".

Post by Shashidhara Shamaiah

Please guide me further

Thanks and Regards
Shashidhara

-----Original Message-----
From: crash-utility-***@redhat.com [mailto:crash-utility-***@redhat.com] On Behalf Of Dave Anderson
Sent: Wednesday, June 15, 2011 8:38 PM
To: Discussion list for crash utility usage,maintenance and development
Subject: Re: [Crash-utility] Unable to switch stack frames while using crash



----- Original Message -----

Post by Shashidhara Shamaiah
Hi,
I was investigating a 64 bit linux kernel dump . I have following
doubts regarding usage of crash.
1) I wanted to access the intermediate kernel stack frames. To know
the status of the frame and the point of failure.
prohibited gdb command: frame ”. Can you please let me know if there
is any other way of accessing the kernel stack frames using crash.

Right -- the embedded gdb doesn't know anything about the core file
(or live system) that you're running on. It's invoked as "gdb vmlinux",
and doesn't know anything about any "frames".

As Flavio mentioned, you can see the stack data of each frame with
"bt -f", or better yet, "bt -F" which may illuminate what the data
may be, because it shows symbolic translations or slab cache names
instead of raw values where appropriate.

Post by Shashidhara Shamaiah
2) When I run bt in crash, I get a stack trace. Another person from a
different team reported a slightly different stack trace to mine.
Below are the stack traces. The register contents are quite different
between the two
My stack trace
PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"
#0 [ffff88031ce759d0] machine_kexec at ffffffff81024486
#1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230
#2 [ffff88031ce75b20] oops_end at ffffffff8100fa38
#3 [ffff88031ce75b50] no_context at ffffffff8102d801
#4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9
#5 [ffff88031ce75c70] bad_area at ffffffff8102da41
#6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19
#7 [ffff88031ce75cf0] page_fault at ffffffff812d7425
#8 [ffff88031ce75d78] n_tty_read at ffffffff811f03b3
#9 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e
#10 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8
#11 [ffff88031ce75f40] sys_read at ffffffff810ebe48
#12 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2
RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212
RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000
RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000
RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
Reported stack trace
PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"
#0 [ffff88031ce759d0] machine_kexec at ffffffff81024486
#1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230
#2 [ffff88031ce75ad8] n_tty_read at ffffffff811f03b3
#3 [ffff88031ce75b20] oops_end at ffffffff8100fa38
#4 [ffff88031ce75b50] no_context at ffffffff8102d801
#5 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9
#6 [ffff88031ce75c20] native_sched_clock at ffffffff810120aa
#7 [ffff88031ce75c70] bad_area at ffffffff8102da41
#8 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19
#9 [ffff88031ce75cf0] page_fault at ffffffff812d7425
[exception RIP: n_tty_read+1420]
RIP: ffffffff811f03b3 RSP: ffff88031ce75da8 RFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044
RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c
RBP: ffff88031ce75eb8 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044
R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #10 [ffff88031ce75ec0]
#10 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e
#11 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8
#12 [ffff88031ce75f40] sys_read at ffffffff810ebe48
#13 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2

The first backtrace is different because you are apparently using an
older version of the crash utility, because it is not showing the
page fault exception frame like the "reported" version.

Post by Shashidhara Shamaiah
3) I want to retrieve the address of a data structure in the current
context. How can it be done? I tried using struct command, but it did
not help

The struct command needs the correct virtual address of the structure
you're trying to view. So I presume you're asking how to find the address
of the data structure? If that's true, you're going to have to be a lot
more specific.

Post by Shashidhara Shamaiah
4) When I run the command readelf -a vmcore, I get an error message
”readelf: Error: Not an ELF file - it has the wrong magic bytes at the
start.”

I presume that the dumpfile is a compressed kdump dumpfile generated
by makedumpfile, which takes the original /proc/vmcore ELF dumpfile
and creates its own unique dumpfile format.

Dave


--
Crash-utility mailing list
Crash-***@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Information transmitted by this e-mail is proprietary to MphasiS, its associated companies and/ or its customers and is intended
for use only by the individual or entity to which it is addressed, and may contain information that is privileged, confidential or
exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded
to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly
prohibited. In such cases, please notify us immediately at ***@mphasis.com and delete this mail from your records.

[Dave Anderson]

----- Original Message -----

Post by Shashidhara Shamaiah
Hi Dave,
Thanks for the help, I have further input regarding query 3 . Please
help.

Post by Shashidhara Shamaiah
3) I want to retrieve the address of a data structure in the current
context. How can it be done? I tried using struct command, but it did
not help

The struct command needs the correct virtual address of the structure
you're trying to view. So I presume you're asking how to find the address
of the data structure? If that's true, you're going to have to be a lot
more specific.

Post by Shashidhara Shamaiah

I need to find out the virtual address of the structure tty of type
struct tty_struct, which is passed as an argument to the function
n_read_tty. Below is the corresponding stack trace.
PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"
#0 [ffff88031ce759d0] machine_kexec at ffffffff81024486
#1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230
#2 [ffff88031ce75b20] oops_end at ffffffff8100fa38
#3 [ffff88031ce75b50] no_context at ffffffff8102d801
#4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9
#5 [ffff88031ce75c70] bad_area at ffffffff8102da41
#6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19
#7 [ffff88031ce75cf0] page_fault at ffffffff812d7425
#8 [ffff88031ce75d78] n_tty_read at ffffffff811f03b3
#9 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e
#10 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8
#11 [ffff88031ce75f40] sys_read at ffffffff810ebe48
#12 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2
RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212
RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000
RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000
RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b

First I would update your crash utility so that you have the exception
frame dump that was a result of the page fault, because it's possible that
the tty structure pointer is in the register dump. But anyway, without
knowing the kernel version, it's hard to pinpoint exactly which instruction
in n_tty_read() generated the page fault. Was the bad address generated
because the tty structure pointer was NULL? And again, with an updated
crash utility, you'll get more information w/respect to the register
contents at the time of the page fault, and also you might get some help
finding it with "bt -F". I'm not sure where the tty structure gets
allocated from -- is it statically-allocated, or is it allocated from
one of the "size-xxx" slab caches, etc...

Post by Shashidhara Shamaiah

Post by Shashidhara Shamaiah

I have another query I tried to convert the vmcore file to ELF
format using "makedumpfile -E -d 31 -x vmlinux_temp vmcore
dumpfile" . For which I got an error message " '-E' option is
disable, because vmcore is kdump compressed format. makedumpfile Failed".
Please guide me further

Refiltering and the -E argument cannot be used together because
makedumpfile cannot regenerate an ELF vmcore file from a previously
compressed kdump dumpfile.

I believe that something like this might work?:

$ makedumpfile -c -d 31 -x vmlinux_temp vmcore-old vmcore-new

Are you trying to re-create an ELF style dumpfile on purpose?

Dave

Post by Shashidhara Shamaiah
Thanks and Regards
Shashidhara
-----Original Message-----
Sent: Wednesday, June 15, 2011 8:38 PM
To: Discussion list for crash utility usage,maintenance and
development
Subject: Re: [Crash-utility] Unable to switch stack frames while using
crash
----- Original Message -----

Post by Shashidhara Shamaiah
Hi,
I was investigating a 64 bit linux kernel dump . I have following
doubts regarding usage of crash.
1) I wanted to access the intermediate kernel stack frames. To know
the status of the frame and the point of failure.
prohibited gdb command: frame ”. Can you please let me know if there
is any other way of accessing the kernel stack frames using crash.

Right -- the embedded gdb doesn't know anything about the core file
(or live system) that you're running on. It's invoked as "gdb
vmlinux",
and doesn't know anything about any "frames".
As Flavio mentioned, you can see the stack data of each frame with
"bt -f", or better yet, "bt -F" which may illuminate what the data
may be, because it shows symbolic translations or slab cache names
instead of raw values where appropriate.

Post by Shashidhara Shamaiah
2) When I run bt in crash, I get a stack trace. Another person from
a
different team reported a slightly different stack trace to mine.
Below are the stack traces. The register contents are quite
different
between the two
My stack trace
PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"
#0 [ffff88031ce759d0] machine_kexec at ffffffff81024486
#1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230
#2 [ffff88031ce75b20] oops_end at ffffffff8100fa38
#3 [ffff88031ce75b50] no_context at ffffffff8102d801
#4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9
#5 [ffff88031ce75c70] bad_area at ffffffff8102da41
#6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19
#7 [ffff88031ce75cf0] page_fault at ffffffff812d7425
#8 [ffff88031ce75d78] n_tty_read at ffffffff811f03b3
#9 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e
#10 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8
#11 [ffff88031ce75f40] sys_read at ffffffff810ebe48
#12 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2
RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212
RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000
RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000
RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
Reported stack trace
PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"
#0 [ffff88031ce759d0] machine_kexec at ffffffff81024486
#1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230
#2 [ffff88031ce75ad8] n_tty_read at ffffffff811f03b3
#3 [ffff88031ce75b20] oops_end at ffffffff8100fa38
#4 [ffff88031ce75b50] no_context at ffffffff8102d801
#5 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9
#6 [ffff88031ce75c20] native_sched_clock at ffffffff810120aa
#7 [ffff88031ce75c70] bad_area at ffffffff8102da41
#8 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19
#9 [ffff88031ce75cf0] page_fault at ffffffff812d7425
[exception RIP: n_tty_read+1420]
RIP: ffffffff811f03b3 RSP: ffff88031ce75da8 RFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044
RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c
RBP: ffff88031ce75eb8 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044
R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #10 [ffff88031ce75ec0]
#10 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e
#11 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8
#12 [ffff88031ce75f40] sys_read at ffffffff810ebe48
#13 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2

The first backtrace is different because you are apparently using an
older version of the crash utility, because it is not showing the
page fault exception frame like the "reported" version.

Post by Shashidhara Shamaiah
3) I want to retrieve the address of a data structure in the current
context. How can it be done? I tried using struct command, but it
did
not help

The struct command needs the correct virtual address of the structure
you're trying to view. So I presume you're asking how to find the
address
of the data structure? If that's true, you're going to have to be a
lot
more specific.

Post by Shashidhara Shamaiah
4) When I run the command readelf -a vmcore, I get an error message
”readelf: Error: Not an ELF file - it has the wrong magic bytes at
the
start.”

I presume that the dumpfile is a compressed kdump dumpfile generated
by makedumpfile, which takes the original /proc/vmcore ELF dumpfile
and creates its own unique dumpfile format.
Dave
--
Crash-utility mailing list
https://www.redhat.com/mailman/listinfo/crash-utility
Information transmitted by this e-mail is proprietary to MphasiS, its
associated companies and/ or its customers and is intended
for use only by the individual or entity to which it is addressed, and
may contain information that is privileged, confidential or
exempt from disclosure under applicable law. If you are not the
intended recipient or it appears that this mail has been forwarded
to you without proper authority, you are notified that any use or
dissemination of this information in any manner is strictly
prohibited. In such cases, please notify us immediately at
--
Crash-utility mailing list
https://www.redhat.com/mailman/listinfo/crash-utility

[Shashidhara Shamaiah]

Hi,

Thank you Dave for your time and help.

As suggested, I will update my crash utility first and then go about analyzing the dump.

Post by Shashidhara Shamaiah

Post by Shashidhara Shamaiah

Post by Dave Anderson
$ makedumpfile -c -d 31 -x vmlinux_temp vmcore-old vmcore-new

I tried to use the command you suggested " makedumpfile -c -d 31 -x vmlinux_temp vmcore vmcore-new " . I got an error message " The kernel version is not supported.The created dumpfile may be incomplete. check_release: Can't get the kernel version"

Should I update makedumpfile utility as well? Or just updating crash will do?

Post by Shashidhara Shamaiah

Post by Shashidhara Shamaiah

Post by Dave Anderson
Are you trying to re-create an ELF style dumpfile on purpose?

I tried to recreate the vmcore file in ELF format because, I can't get access to the original uncompressed ELF dump file which is in the customer machine.

Thanks and Regards
Shashidhara


-----Original Message-----
From: crash-utility-***@redhat.com [mailto:crash-utility-***@redhat.com] On Behalf Of Dave Anderson
Sent: Wednesday, June 15, 2011 9:33 PM
To: Discussion list for crash utility usage,maintenance and development
Subject: Re: [Crash-utility] Unable to switch stack frames while using crash



----- Original Message -----

Post by Shashidhara Shamaiah
Hi Dave,
Thanks for the help, I have further input regarding query 3 . Please help.

Post by Shashidhara Shamaiah
3) I want to retrieve the address of a data structure in the current
context. How can it be done? I tried using struct command, but it did
not help

The struct command needs the correct virtual address of the structure
you're trying to view. So I presume you're asking how to find the address
of the data structure? If that's true, you're going to have to be a lot
more specific.

Post by Shashidhara Shamaiah

Post by Dave Anderson
I need to find out the virtual address of the structure tty of type
struct tty_struct, which is passed as an argument to the function
n_read_tty. Below is the corresponding stack trace.
PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"
#0 [ffff88031ce759d0] machine_kexec at ffffffff81024486
#1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230
#2 [ffff88031ce75b20] oops_end at ffffffff8100fa38
#3 [ffff88031ce75b50] no_context at ffffffff8102d801
#4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9
#5 [ffff88031ce75c70] bad_area at ffffffff8102da41
#6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19
#7 [ffff88031ce75cf0] page_fault at ffffffff812d7425
#8 [ffff88031ce75d78] n_tty_read at ffffffff811f03b3
#9 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e
#10 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8
#11 [ffff88031ce75f40] sys_read at ffffffff810ebe48
#12 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2
RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212
RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000
RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000
RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b

First I would update your crash utility so that you have the exception
frame dump that was a result of the page fault, because it's possible that
the tty structure pointer is in the register dump. But anyway, without
knowing the kernel version, it's hard to pinpoint exactly which instruction
in n_tty_read() generated the page fault. Was the bad address generated
because the tty structure pointer was NULL? And again, with an updated
crash utility, you'll get more information w/respect to the register
contents at the time of the page fault, and also you might get some help
finding it with "bt -F". I'm not sure where the tty structure gets
allocated from -- is it statically-allocated, or is it allocated from
one of the "size-xxx" slab caches, etc...

Post by Shashidhara Shamaiah

Post by Shashidhara Shamaiah

Post by Dave Anderson
I have another query I tried to convert the vmcore file to ELF
format using "makedumpfile -E -d 31 -x vmlinux_temp vmcore
dumpfile" . For which I got an error message " '-E' option is
disable, because vmcore is kdump compressed format. makedumpfile Failed".
Please guide me further

Refiltering and the -E argument cannot be used together because
makedumpfile cannot regenerate an ELF vmcore file from a previously
compressed kdump dumpfile.

I believe that something like this might work?:

$ makedumpfile -c -d 31 -x vmlinux_temp vmcore-old vmcore-new

Are you trying to re-create an ELF style dumpfile on purpose?

Dave

Post by Shashidhara Shamaiah
Thanks and Regards
Shashidhara
-----Original Message-----
Sent: Wednesday, June 15, 2011 8:38 PM
To: Discussion list for crash utility usage,maintenance and
development
Subject: Re: [Crash-utility] Unable to switch stack frames while using crash
----- Original Message -----

Post by Shashidhara Shamaiah
Hi,
I was investigating a 64 bit linux kernel dump . I have following
doubts regarding usage of crash.
1) I wanted to access the intermediate kernel stack frames. To know
the status of the frame and the point of failure.
prohibited gdb command: frame ”. Can you please let me know if there
is any other way of accessing the kernel stack frames using crash.

Right -- the embedded gdb doesn't know anything about the core file
(or live system) that you're running on. It's invoked as "gdb
vmlinux",
and doesn't know anything about any "frames".
As Flavio mentioned, you can see the stack data of each frame with
"bt -f", or better yet, "bt -F" which may illuminate what the data
may be, because it shows symbolic translations or slab cache names
instead of raw values where appropriate.

Post by Shashidhara Shamaiah
2) When I run bt in crash, I get a stack trace. Another person from a
different team reported a slightly different stack trace to mine.
Below are the stack traces. The register contents are quite
different
between the two
My stack trace
PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"
#0 [ffff88031ce759d0] machine_kexec at ffffffff81024486
#1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230
#2 [ffff88031ce75b20] oops_end at ffffffff8100fa38
#3 [ffff88031ce75b50] no_context at ffffffff8102d801
#4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9
#5 [ffff88031ce75c70] bad_area at ffffffff8102da41
#6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19
#7 [ffff88031ce75cf0] page_fault at ffffffff812d7425
#8 [ffff88031ce75d78] n_tty_read at ffffffff811f03b3
#9 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e
#10 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8
#11 [ffff88031ce75f40] sys_read at ffffffff810ebe48
#12 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2
RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212
RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000
RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000
RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
Reported stack trace
PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"
#0 [ffff88031ce759d0] machine_kexec at ffffffff81024486
#1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230
#2 [ffff88031ce75ad8] n_tty_read at ffffffff811f03b3
#3 [ffff88031ce75b20] oops_end at ffffffff8100fa38
#4 [ffff88031ce75b50] no_context at ffffffff8102d801
#5 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9
#6 [ffff88031ce75c20] native_sched_clock at ffffffff810120aa
#7 [ffff88031ce75c70] bad_area at ffffffff8102da41
#8 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19
#9 [ffff88031ce75cf0] page_fault at ffffffff812d7425
[exception RIP: n_tty_read+1420]
RIP: ffffffff811f03b3 RSP: ffff88031ce75da8 RFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044
RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c
RBP: ffff88031ce75eb8 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044
R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #10 [ffff88031ce75ec0]
#10 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e
#11 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8
#12 [ffff88031ce75f40] sys_read at ffffffff810ebe48
#13 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2

The first backtrace is different because you are apparently using an
older version of the crash utility, because it is not showing the
page fault exception frame like the "reported" version.

Post by Shashidhara Shamaiah
3) I want to retrieve the address of a data structure in the current
context. How can it be done? I tried using struct command, but it did
not help

The struct command needs the correct virtual address of the structure
you're trying to view. So I presume you're asking how to find the address
of the data structure? If that's true, you're going to have to be a lot
more specific.

Post by Shashidhara Shamaiah
4) When I run the command readelf -a vmcore, I get an error message
”readelf: Error: Not an ELF file - it has the wrong magic bytes at the
start.”

I presume that the dumpfile is a compressed kdump dumpfile generated
by makedumpfile, which takes the original /proc/vmcore ELF dumpfile
and creates its own unique dumpfile format.
Dave
--
Crash-utility mailing list
https://www.redhat.com/mailman/listinfo/crash-utility
Information transmitted by this e-mail is proprietary to MphasiS, its
associated companies and/ or its customers and is intended
for use only by the individual or entity to which it is addressed, and
may contain information that is privileged, confidential or
exempt from disclosure under applicable law. If you are not the
intended recipient or it appears that this mail has been forwarded
to you without proper authority, you are notified that any use or
dissemination of this information in any manner is strictly
prohibited. In such cases, please notify us immediately at
--
Crash-utility mailing list
https://www.redhat.com/mailman/listinfo/crash-utility

--
Crash-utility mailing list
Crash-***@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Information transmitted by this e-mail is proprietary to MphasiS, its associated companies and/ or its customers and is intended
for use only by the individual or entity to which it is addressed, and may contain information that is privileged, confidential or
exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded
to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly
prohibited. In such cases, please notify us immediately at ***@mphasis.com and delete this mail from your records.

[Dave Anderson]

----- Original Message -----

Post by Dave Anderson
First I would update your crash utility so that you have the exception
frame dump that was a result of the page fault, because it's possible that
the tty structure pointer is in the register dump. But anyway, without
knowing the kernel version, it's hard to pinpoint exactly which instruction
in n_tty_read() generated the page fault. Was the bad address generated
because the tty structure pointer was NULL? And again, with an updated
crash utility, you'll get more information w/respect to the register
contents at the time of the page fault, and also you might get some help
finding it with "bt -F". I'm not sure where the tty structure gets
allocated from -- is it statically-allocated, or is it allocated from
one of the "size-xxx" slab caches, etc...

BTW, looking at the other guy's report, whose backtrace did contain
the page fault exception frame, you can see that the page fault was
generated upon the execution of the instruction at ffffffff811f03b3,
which is n_tty_read+1420:

...
#9 [ffff88031ce75cf0] page_fault at ffffffff812d7425
[exception RIP: n_tty_read+1420]
RIP: ffffffff811f03b3 RSP: ffff88031ce75da8 RFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044
RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c
RBP: ffff88031ce75eb8 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044
R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#10 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e
...

To find out the line of code that generated the page fault, enter this:

crash> dis -rl n_tty_read+1420

The disassembly will start at the beginning of n_tty_read() and stop at
the instruction above that actually caused the page fault, and you will
also see the source-file/line-number information above that.

I checked a few sample kernels, but none of them seem to have a fault-able
instruction exactly at the exception RIP of n_tty_read+1420, but I'm sure
that if you look at your particular kernel source tree, it will make sense.

Dave

[Dave Anderson]

----- Original Message -----

Post by Shashidhara Shamaiah
Hi,
Thank you Dave for your time and help.
As suggested, I will update my crash utility first and then go about
analyzing the dump.

Post by Dave Anderson
$ makedumpfile -c -d 31 -x vmlinux_temp vmcore-old vmcore-new

I tried to use the command you suggested " makedumpfile -c -d 31 -x
vmlinux_temp vmcore vmcore-new " . I got an error message " The kernel
version is not supported.The created dumpfile may be incomplete.
check_release: Can't get the kernel version"
Should I update makedumpfile utility as well? Or just updating crash
will do?

That's a makedumpfile issue -- the message comes from here in makedumpfile.c:

if ((version < OLDEST_VERSION) || (LATEST_VERSION < version)) {
MSG("The kernel version is not supported.\n");
MSG("The created dumpfile may be incomplete.\n");
}

where, at least in the RHEL6 version of makedumpfile (1.3.5), this
is the range:

#define OLDEST_VERSION KERNEL_VERSION(2, 6, 15)/* linux-2.6.15 */
#define LATEST_VERSION KERNEL_VERSION(2, 6, 32)/* linux-2.6.32 */

The upstream version of makedumpfile is now 1.3.7, which bumps LATEST_VERSION:

#define LATEST_VERSION KERNEL_VERSION(2, 6, 36)/* linux-2.6.36 */

But it's a warning message, and your vmcore-new may still be usable.

Dave

[Shashidhara Shamaiah]

Hi Dave,

I updated the makedumpfile utility from 1.3.5 to 1.3.7 . When I run the
below command

makedumpfile -c -d 31 -x vmlinux_temp vmcore vmcore-new
The kernel version is not supported.
The created dumpfile may be incomplete.
check_release: Can't get the kernel version.
makedumpfile Failed.

Is there any other way to extract the ELF style vmcore file from the
kdump compressed format. Please guide me.


Regards
Shashidhara

-----Original Message-----
From: crash-utility-***@redhat.com
[mailto:crash-utility-***@redhat.com] On Behalf Of Dave Anderson
Sent: Thursday, June 16, 2011 7:30 PM
To: Discussion list for crash utility usage,maintenance and development
Subject: Re: [Crash-utility] Unable to switch stack frames while using
crash



----- Original Message -----

Post by Shashidhara Shamaiah
Hi,
Thank you Dave for your time and help.
As suggested, I will update my crash utility first and then go about analyzing the dump.

Post by Dave Anderson
$ makedumpfile -c -d 31 -x vmlinux_temp vmcore-old vmcore-new

I tried to use the command you suggested " makedumpfile -c -d 31 -x
vmlinux_temp vmcore vmcore-new " . I got an error message " The kernel
version is not supported.The created dumpfile may be incomplete.
check_release: Can't get the kernel version"
Should I update makedumpfile utility as well? Or just updating crash will do?

That's a makedumpfile issue -- the message comes from here in
makedumpfile.c:

if ((version < OLDEST_VERSION) || (LATEST_VERSION < version)) {
MSG("The kernel version is not supported.\n");
MSG("The created dumpfile may be incomplete.\n");
}

where, at least in the RHEL6 version of makedumpfile (1.3.5), this
is the range:

#define OLDEST_VERSION KERNEL_VERSION(2, 6, 15)/*
linux-2.6.15 */
#define LATEST_VERSION KERNEL_VERSION(2, 6, 32)/*
linux-2.6.32 */

The upstream version of makedumpfile is now 1.3.7, which bumps
LATEST_VERSION:

#define LATEST_VERSION KERNEL_VERSION(2, 6, 36)/*
linux-2.6.36 */

But it's a warning message, and your vmcore-new may still be usable.

Dave



--
Crash-utility mailing list
Crash-***@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Information transmitted by this e-mail is proprietary to MphasiS, its associated companies and/ or its customers and is intended
for use only by the individual or entity to which it is addressed, and may contain information that is privileged, confidential or
exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded
to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly
prohibited. In such cases, please notify us immediately at ***@mphasis.com and delete this mail from your records.

[Dave Anderson]

----- Original Message -----

Post by Shashidhara Shamaiah
Hi Dave,
I updated the makedumpfile utility from 1.3.5 to 1.3.7 . When I run the
below command
makedumpfile -c -d 31 -x vmlinux_temp vmcore vmcore-new
The kernel version is not supported.
The created dumpfile may be incomplete.
check_release: Can't get the kernel version.
makedumpfile Failed.

I see that makedumpfile-1.3.8 was recently released, but it still
has a LATEST_VERSION of 2.6.36:

#define OLDEST_VERSION KERNEL_VERSION(2, 6, 15)/* linux-2.6.15 */
#define LATEST_VERSION KERNEL_VERSION(2, 6, 36)/* linux-2.6.36 */

You haven't stated what your kernel version is, but it seems makedumpfile
cannot get past this point. On the other hand, the compressed kdump was
created, so I'm not entirely clear.

Post by Shashidhara Shamaiah
Is there any other way to extract the ELF style vmcore file from the
kdump compressed format. Please guide me.

I don't believe so...

But I'm not the makedumpfile maintainer, so I'd prefer not to give any
definitive answers to your questions. I've cc'd the upstream maintainer
of makedumpfile.

Thanks,
Dave

[Shashidhara Shamaiah]

Hi Dave,

Thanks for the help.

I have some doubts regarding kdump and crash utility.

I am analyzing a vmcore dump caused by an oops in customer location
using crash utility.The oops report is below

[345132.723424] BUG: unable to handle kernel NULL pointer dereference at
0000000000000005
[345132.724928] IP: [<ffffffff811f03b3>] n_tty_read+0x58c/0x818
[345132.726100] PGD 2c8e03067 PUD 2cbd88067 PMD 0
[345132.727187] Oops: 0000 [#1] SMP
[345132.727879] last sysfs file: /sys/block/loop7/dev
[345132.728935] CPU 1
[345132.729396] Modules linked in: xt_tcpudp iptable_filter ip_tables
x_tables strmfs_mod bond0 ipmi_devintf hpwdt sctp ipv6 crc32c libcrc32c
loop ipmi_si tpm_tis ipmi_msghandler hpilo tpm tpm_bios psmouse
serio_raw shpchp pci_hotplug container processor evdev ext3 jbd mbcache
dm_mirror dm_region_hash dm_log dm_snapshot dm_mod sg sr_mod cdrom
ide_pci_generic ide_core usbhid hid ata_piix ata_generic libata ehci_hcd
bnx2 uhci_hcd e1000e cciss scsi_mod button thermal fan thermal_sys edd
[last unloaded: scsi_wait_scan]
[345132.739511] Pid: 13366, comm: telnet Not tainted
2.6.32-cdma-18-amd64 #1 ProLiant DL380 G6
[345132.741423] RIP: 0010:[<ffffffff811f03b3>] [<ffffffff811f03b3>]
n_tty_read+0x58c/0x818
[345132.743220] RSP: 0018:ffff88031ce75da8 EFLAGS: 00010246
[345132.744469] RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX:
000000000061c044
[345132.746061] RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI:
ffff8802cbd54d1c
[345132.747726] RBP: ffff88031ce75eb8 R08: 0000000000000000 R09:
0000000000000000
[345132.749391] R10: 0000000000616680 R11: 0000000000000246 R12:
000000000061c044
[345132.750981] R13: ffff8802cbd54800 R14: 0000000000000000 R15:
7fffffffffffffff
[345132.752650] FS: 00007ffff7fee6f0(0000) GS:ffff880033020000(0000)
knlGS:0000000000000000
[345132.754569] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[345132.755915] CR2: 0000000000000005 CR3: 000000030c408000 CR4:
00000000000006e0
[345132.757579] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[345132.759169] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[345132.760778] Process telnet (pid: 13366, threadinfo ffff88031ce74000,
task ffff88031b60d580)
[345132.762707] Stack:
[345132.763162] ffff88031b60d580 ffff88031b60d580 ffff88031b60d580
ffff88031b60d580
[345132.764791] <0> 000000000061c02b 0000000000000000 0000000000000000
000000000061c02a
[345132.766510] <0> ffff8802de651a40 ffff8802cbd549c0 ffff8802cbd54c90
ffff8802cbd54d1c
[345132.768270] Call Trace:
[345132.768877] [<ffffffff81045f84>] ? default_wake_function+0x0/0xf
[345132.770309] [<ffffffff811ebf7e>] tty_read+0x7d/0xba
[345132.771526] [<ffffffff810ebcc8>] vfs_read+0xab/0x167
[345132.772541] [<ffffffff810ebe48>] sys_read+0x47/0x6f
[345132.773526] [<ffffffff8100bbc2>] system_call_fastpath+0x16/0x1b
[345132.774652] Code: 00 41 8b 85 5c 02 00 00 48 8b 9d 78 ff ff ff f0 0f
b3 03 45 19 f6 49 63 95 5c 02 00 00 49 8b 85 50 02 00 00 48 8b bd 48 ff
ff ff <0f> be 1c 10 e8 fc 6b 0e 00 48 89 c6 41 8b 85 5c 02 00 00 41 ff
[345132.778840] RIP [<ffffffff811f03b3>] n_tty_read+0x58c/0x818
[345132.780107] RSP <ffff88031ce75da8>
[345132.780969] CR2: 0000000000000005
[345132.781786] hpwdt: New timer passed in is 120 seconds.
[345132.782942] hpwdt: timer reset to 120 for kdump


After analysis, we figured out that the crash occurs in the function
n_read_tty of kernel-source/drivers/char/n_tty.c . The oops occurred on
linux kernel 2.6.32. Below is the code fragment where the page fault
occurred. The page fault occurs when executing the statement c =
tty->read_buf[tty->read_tail] .

/* N.B. avoid overrun if nr == 0 */
while (nr && tty->read_cnt) {

int eol;

eol = test_and_clear_bit(tty->read_tail,
tty->read_flags);
c = tty->read_buf[tty->read_tail]; //
page fault statement after analyzing oops


spin_lock_irqsave(&tty->read_lock, flags);
tty->read_tail = ((tty->read_tail+1) &
(N_TTY_BUF_SIZE-1));
tty->read_cnt--;
if (eol) {
/* this test should be
redundant:
* we shouldn't be reading data
if
* canon_data is 0
*/
if (--tty->canon_data < 0)
tty->canon_data = 0;
}
spin_unlock_irqrestore(&tty->read_lock,
flags);




Below is the contents of the structure tty_struct ( at the time of oops
). This was passed as an argument to the function n_read_tty().

tty_struct ffff8802cbd54800
struct tty_struct { ...
magic = 21505,
driver = 0xffff88031b54ea00,
ops = 0xffffffff8130f650,
name = "pts9\000\...",
driver_data = 0xffff88029c8a9668,
icanon = 1 '\001',
read_buf = 0xffff8802cbfe6000 "",
read_head = 0,
read_tail = 0,
read_cnt = 0,
read_flags = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0},
canon_data = 0,
......................................

As per crash utility the field read_cnt is 0 when kernel oopsed.In that
case, the statement while (nr && tty->read_cnt) in the above code
fragment should have failed. This leads me to think that there was some
other thread/task in kernel which should have updated the read_cnt
field in parallel. However the crash utility reports that the runqueue
of all CPUs at the time of crash as idle. Except CPU1 which was
executing the user program telnet in kernel context ( system call ).
Below is the runqueue output.

CPU 0 RUNQUEUE: ffff880033012d80
CURRENT: PID: 0 TASK: ffffffff814204b0 COMMAND: "swapper"
RT PRIO_ARRAY: ffff880033012e98
[no tasks queued]
CFS RB_ROOT: ffff880033012e10
[no tasks queued]

CPU 1 RUNQUEUE: ffff880033032d80
CURRENT: PID: 13366 TASK: ffff88031b60d580 COMMAND: "telnet"
RT PRIO_ARRAY: ffff880033032e98
[no tasks queued]
CFS RB_ROOT: ffff880033032e10
[no tasks queued]

CPU 2 RUNQUEUE: ffff880033052d80
CURRENT: PID: 0 TASK: ffff88031e0e3540 COMMAND: "swapper"
RT PRIO_ARRAY: ffff880033052e98
[no tasks queued]
CFS RB_ROOT: ffff880033052e10
[no tasks queued]

CPU 3 RUNQUEUE: ffff880033072d80
CURRENT: PID: 0 TASK: ffff88031e113580 COMMAND: "swapper"
RT PRIO_ARRAY: ffff880033072e98
[no tasks queued]
CFS RB_ROOT: ffff880033072e10
[no tasks queued]


How is this logically possible. Crash reports there are no tasks running
currently. Or before the oops trigger and kdump capturing the memory
image, some process/thread ran which could have updated the data
structure. I wanted to know if this scenario is possible. I kindly
request your suggestion/guidance. Please let me know if you need any
other details.


Regards
Shashidhara


-----Original Message-----
From: crash-utility-***@redhat.com
[mailto:crash-utility-***@redhat.com] On Behalf Of Dave Anderson
Sent: Tuesday, June 21, 2011 7:24 PM
To: Discussion list for crash utility usage,maintenance and development
Subject: Re: [Crash-utility] Unable to switch stack frames while using
crash



----- Original Message -----

Post by Shashidhara Shamaiah
Hi Dave,
I updated the makedumpfile utility from 1.3.5 to 1.3.7 . When I run the
below command
makedumpfile -c -d 31 -x vmlinux_temp vmcore vmcore-new
The kernel version is not supported.
The created dumpfile may be incomplete.
check_release: Can't get the kernel version.
makedumpfile Failed.

I see that makedumpfile-1.3.8 was recently released, but it still
has a LATEST_VERSION of 2.6.36:

#define OLDEST_VERSION KERNEL_VERSION(2, 6, 15)/*
linux-2.6.15 */
#define LATEST_VERSION KERNEL_VERSION(2, 6, 36)/*
linux-2.6.36 */

You haven't stated what your kernel version is, but it seems
makedumpfile
cannot get past this point. On the other hand, the compressed kdump was
created, so I'm not entirely clear.

Post by Shashidhara Shamaiah
Is there any other way to extract the ELF style vmcore file from the
kdump compressed format. Please guide me.

I don't believe so...

But I'm not the makedumpfile maintainer, so I'd prefer not to give any
definitive answers to your questions. I've cc'd the upstream maintainer
of makedumpfile.

Thanks,
Dave

--
Crash-utility mailing list
Crash-***@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Information transmitted by this e-mail is proprietary to MphasiS, its associated companies and/ or its customers and is intended
for use only by the individual or entity to which it is addressed, and may contain information that is privileged, confidential or
exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded
to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly
prohibited. In such cases, please notify us immediately at ***@mphasis.com and delete this mail from your records.

[Dave Anderson]

----- Original Message -----

Post by Shashidhara Shamaiah
Hi Dave,
Thanks for the help.
I have some doubts regarding kdump and crash utility.
I am analyzing a vmcore dump caused by an oops in customer location
using crash utility.The oops report is below
[345132.723424] BUG: unable to handle kernel NULL pointer dereference
at
0000000000000005
[345132.724928] IP: [<ffffffff811f03b3>] n_tty_read+0x58c/0x818
[345132.726100] PGD 2c8e03067 PUD 2cbd88067 PMD 0
[345132.727187] Oops: 0000 [#1] SMP
[345132.727879] last sysfs file: /sys/block/loop7/dev
[345132.728935] CPU 1
[345132.729396] Modules linked in: xt_tcpudp iptable_filter ip_tables
x_tables strmfs_mod bond0 ipmi_devintf hpwdt sctp ipv6 crc32c
libcrc32c
loop ipmi_si tpm_tis ipmi_msghandler hpilo tpm tpm_bios psmouse
serio_raw shpchp pci_hotplug container processor evdev ext3 jbd
mbcache
dm_mirror dm_region_hash dm_log dm_snapshot dm_mod sg sr_mod cdrom
ide_pci_generic ide_core usbhid hid ata_piix ata_generic libata
ehci_hcd
bnx2 uhci_hcd e1000e cciss scsi_mod button thermal fan thermal_sys edd
[last unloaded: scsi_wait_scan]
[345132.739511] Pid: 13366, comm: telnet Not tainted
2.6.32-cdma-18-amd64 #1 ProLiant DL380 G6
[345132.741423] RIP: 0010:[<ffffffff811f03b3>] [<ffffffff811f03b3>] n_tty_read+0x58c/0x818
[345132.743220] RSP: 0018:ffff88031ce75da8 EFLAGS: 00010246
[345132.744469] RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044
[345132.746061] RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c
[345132.747726] RBP: ffff88031ce75eb8 R08: 0000000000000000 R09: 0000000000000000
[345132.749391] R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044
[345132.750981] R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff
[345132.752650] FS: 00007ffff7fee6f0(0000) GS:ffff880033020000(0000) knlGS:0000000000000000
[345132.754569] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[345132.755915] CR2: 0000000000000005 CR3: 000000030c408000 CR4: 00000000000006e0
[345132.757579] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[345132.759169] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[345132.760778] Process telnet (pid: 13366, threadinfo ffff88031ce74000, task ffff88031b60d580)
[345132.763162] ffff88031b60d580 ffff88031b60d580 ffff88031b60d580 ffff88031b60d580
[345132.764791] <0> 000000000061c02b 0000000000000000 0000000000000000 000000000061c02a
[345132.766510] <0> ffff8802de651a40 ffff8802cbd549c0 ffff8802cbd54c90 ffff8802cbd54d1c
[345132.768877] [<ffffffff81045f84>] ? default_wake_function+0x0/0xf
[345132.770309] [<ffffffff811ebf7e>] tty_read+0x7d/0xba
[345132.771526] [<ffffffff810ebcc8>] vfs_read+0xab/0x167
[345132.772541] [<ffffffff810ebe48>] sys_read+0x47/0x6f
[345132.773526] [<ffffffff8100bbc2>] system_call_fastpath+0x16/0x1b
[345132.774652] Code: 00 41 8b 85 5c 02 00 00 48 8b 9d 78 ff ff ff f0 0f
b3 03 45 19 f6 49 63 95 5c 02 00 00 49 8b 85 50 02 00 00 48 8b bd 48 ff
ff ff <0f> be 1c 10 e8 fc 6b 0e 00 48 89 c6 41 8b 85 5c 02 00 00 41 ff
[345132.778840] RIP [<ffffffff811f03b3>] n_tty_read+0x58c/0x818
[345132.780107] RSP <ffff88031ce75da8>
[345132.780969] CR2: 0000000000000005
[345132.781786] hpwdt: New timer passed in is 120 seconds.
[345132.782942] hpwdt: timer reset to 120 for kdump
After analysis, we figured out that the crash occurs in the function
n_read_tty of kernel-source/drivers/char/n_tty.c . The oops occurred on
linux kernel 2.6.32. Below is the code fragment where the page fault
occurred. The page fault occurs when executing the statement c =
tty->read_buf[tty->read_tail] .
/* N.B. avoid overrun if nr == 0 */
while (nr && tty->read_cnt) {
int eol;
eol = test_and_clear_bit(tty->read_tail,
tty->read_flags);
c = tty->read_buf[tty->read_tail]; //
page fault statement after analyzing oops
spin_lock_irqsave(&tty->read_lock, flags);
tty->read_tail = ((tty->read_tail+1) &
(N_TTY_BUF_SIZE-1));
tty->read_cnt--;
if (eol) {
/* this test should be
* we shouldn't be reading data
if
* canon_data is 0
*/
if (--tty->canon_data < 0)
tty->canon_data = 0;
}
spin_unlock_irqrestore(&tty->read_lock,
flags);
Below is the contents of the structure tty_struct ( at the time of
oops
). This was passed as an argument to the function n_read_tty().
tty_struct ffff8802cbd54800
struct tty_struct { ...
magic = 21505,
driver = 0xffff88031b54ea00,
ops = 0xffffffff8130f650,
name = "pts9\000\...",
driver_data = 0xffff88029c8a9668,
icanon = 1 '\001',
read_buf = 0xffff8802cbfe6000 "",
read_head = 0,
read_tail = 0,
read_cnt = 0,
read_flags = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0},
canon_data = 0,
......................................
As per crash utility the field read_cnt is 0 when kernel oopsed.In that
case, the statement while (nr && tty->read_cnt) in the above code
fragment should have failed. This leads me to think that there was some
other thread/task in kernel which should have updated the read_cnt
field in parallel. However the crash utility reports that the runqueue
of all CPUs at the time of crash as idle. Except CPU1 which was
executing the user program telnet in kernel context ( system call ).
Below is the runqueue output.
CPU 0 RUNQUEUE: ffff880033012d80
CURRENT: PID: 0 TASK: ffffffff814204b0 COMMAND: "swapper"
RT PRIO_ARRAY: ffff880033012e98
[no tasks queued]
CFS RB_ROOT: ffff880033012e10
[no tasks queued]
CPU 1 RUNQUEUE: ffff880033032d80
CURRENT: PID: 13366 TASK: ffff88031b60d580 COMMAND: "telnet"
RT PRIO_ARRAY: ffff880033032e98
[no tasks queued]
CFS RB_ROOT: ffff880033032e10
[no tasks queued]
CPU 2 RUNQUEUE: ffff880033052d80
CURRENT: PID: 0 TASK: ffff88031e0e3540 COMMAND: "swapper"
RT PRIO_ARRAY: ffff880033052e98
[no tasks queued]
CFS RB_ROOT: ffff880033052e10
[no tasks queued]
CPU 3 RUNQUEUE: ffff880033072d80
CURRENT: PID: 0 TASK: ffff88031e113580 COMMAND: "swapper"
RT PRIO_ARRAY: ffff880033072e98
[no tasks queued]
CFS RB_ROOT: ffff880033072e10
[no tasks queued]
How is this logically possible. Crash reports there are no tasks running
currently. Or before the oops trigger and kdump capturing the memory
image, some process/thread ran which could have updated the data
structure. I wanted to know if this scenario is possible. I kindly
request your suggestion/guidance. Please let me know if you need any
other details.

It's not clear to me, but I'm not at all familiar with this code area.
Maybe during hard or soft IRQ handling on this or another cpu? Presumably
there would be protection against that happening, and maybe it's of interest
that the very next instruction after the fault is a spin_lock_irqsave()
call, but that's just a wild guess on my part...

Anyway, the crash utility shows what the state of memory was at the point
when the "telnet" process (indirectly) issued NMI interrupts to all of
the other cpus. You can verify where the other cpus were (in idle) by
"bt -a", which shows/verifies the reception of the NMI shutdown interrupt.

Dave

[Dave Anderson]

----- Original Message -----

Post by Shashidhara Shamaiah
After analysis, we figured out that the crash occurs in the function
n_read_tty of kernel-source/drivers/char/n_tty.c . The oops occurred on
linux kernel 2.6.32. Below is the code fragment where the page fault
occurred. The page fault occurs when executing the statement c =
tty->read_buf[tty->read_tail] .
/* N.B. avoid overrun if nr == 0 */
while (nr && tty->read_cnt) {
int eol;
eol = test_and_clear_bit(tty->read_tail, tty->read_flags);
c = tty->read_buf[tty->read_tail]; //
page fault statement after analyzing oops

BTW, are you sure about that?

Presuming that the "tty" pointer is ffff8802cbd54800 as you've shown below,
and therefore tty->read_buf is 0xffff8802cbfe6000 and tty->read_tail is 0,
then the statement above would be simply be reading tty->read_buf[0], or
virtual address 0xffff8802cbfe6000. But the oops shows it faulting on a
virtual address of "5":

BUG: unable to handle kernel NULL pointer dereference at 0000000000000005

Dave

Post by Shashidhara Shamaiah
Below is the contents of the structure tty_struct ( at the time of
oops
). This was passed as an argument to the function n_read_tty().
tty_struct ffff8802cbd54800
struct tty_struct { ...
magic = 21505,
driver = 0xffff88031b54ea00,
ops = 0xffffffff8130f650,
name = "pts9\000\...",
driver_data = 0xffff88029c8a9668,
icanon = 1 '\001',
read_buf = 0xffff8802cbfe6000 "",
read_head = 0,
read_tail = 0,
read_cnt = 0,
read_flags = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0},
canon_data = 0,
......................................
As per crash utility the field read_cnt is 0 when kernel oopsed.In
that
case, the statement while (nr && tty->read_cnt) in the above code
fragment should have failed. This leads me to think that there was
some
other thread/task in kernel which should have updated the read_cnt
field in parallel. However the crash utility reports that the runqueue
of all CPUs at the time of crash as idle. Except CPU1 which was
executing the user program telnet in kernel context ( system call ).
Below is the runqueue output.
CPU 0 RUNQUEUE: ffff880033012d80
CURRENT: PID: 0 TASK: ffffffff814204b0 COMMAND: "swapper"
RT PRIO_ARRAY: ffff880033012e98
[no tasks queued]
CFS RB_ROOT: ffff880033012e10
[no tasks queued]
CPU 1 RUNQUEUE: ffff880033032d80
CURRENT: PID: 13366 TASK: ffff88031b60d580 COMMAND: "telnet"
RT PRIO_ARRAY: ffff880033032e98
[no tasks queued]
CFS RB_ROOT: ffff880033032e10
[no tasks queued]
CPU 2 RUNQUEUE: ffff880033052d80
CURRENT: PID: 0 TASK: ffff88031e0e3540 COMMAND: "swapper"
RT PRIO_ARRAY: ffff880033052e98
[no tasks queued]
CFS RB_ROOT: ffff880033052e10
[no tasks queued]
CPU 3 RUNQUEUE: ffff880033072d80
CURRENT: PID: 0 TASK: ffff88031e113580 COMMAND: "swapper"
RT PRIO_ARRAY: ffff880033072e98
[no tasks queued]
CFS RB_ROOT: ffff880033072e10
[no tasks queued]
How is this logically possible. Crash reports there are no tasks
running
currently. Or before the oops trigger and kdump capturing the memory
image, some process/thread ran which could have updated the data
structure. I wanted to know if this scenario is possible. I kindly
request your suggestion/guidance. Please let me know if you need any
other details.
Regards
Shashidhara
-----Original Message-----
Sent: Tuesday, June 21, 2011 7:24 PM
To: Discussion list for crash utility usage,maintenance and
development
Subject: Re: [Crash-utility] Unable to switch stack frames while using
crash
----- Original Message -----

Post by Shashidhara Shamaiah
Hi Dave,
I updated the makedumpfile utility from 1.3.5 to 1.3.7 . When I run

the

Post by Shashidhara Shamaiah
below command
makedumpfile -c -d 31 -x vmlinux_temp vmcore vmcore-new
The kernel version is not supported.
The created dumpfile may be incomplete.
check_release: Can't get the kernel version.
makedumpfile Failed.

I see that makedumpfile-1.3.8 was recently released, but it still
#define OLDEST_VERSION KERNEL_VERSION(2, 6, 15)/*
linux-2.6.15 */
#define LATEST_VERSION KERNEL_VERSION(2, 6, 36)/*
linux-2.6.36 */
You haven't stated what your kernel version is, but it seems
makedumpfile
cannot get past this point. On the other hand, the compressed kdump
was
created, so I'm not entirely clear.

Post by Shashidhara Shamaiah
Is there any other way to extract the ELF style vmcore file from the
kdump compressed format. Please guide me.

I don't believe so...
But I'm not the makedumpfile maintainer, so I'd prefer not to give any
definitive answers to your questions. I've cc'd the upstream
maintainer
of makedumpfile.
Thanks,
Dave
--
Crash-utility mailing list
https://www.redhat.com/mailman/listinfo/crash-utility
Information transmitted by this e-mail is proprietary to MphasiS, its
associated companies and/ or its customers and is intended
for use only by the individual or entity to which it is addressed, and
may contain information that is privileged, confidential or
exempt from disclosure under applicable law. If you are not the
intended recipient or it appears that this mail has been forwarded
to you without proper authority, you are notified that any use or
dissemination of this information in any manner is strictly
prohibited. In such cases, please notify us immediately at
--
Crash-utility mailing list
https://www.redhat.com/mailman/listinfo/crash-utility

[Dave Anderson]

----- Original Message -----

Post by Dave Anderson
BTW, are you sure about that?
Presuming that the "tty" pointer is ffff8802cbd54800 as you've shown below,
and therefore tty->read_buf is 0xffff8802cbfe6000 and tty->read_tail is 0,
then the statement above would be simply be reading tty->read_buf[0], or
virtual address 0xffff8802cbfe6000. But the oops shows it faulting on a
BUG: unable to handle kernel NULL pointer dereference at 0000000000000005

Just for my own sanity, can you either attach the "drivers/char/n_tty.c"
from *your* specific kernel, or get the source-code/line-number data from
the embedded gdb module?

If you don't have the n_tty.c file readily available, you can get the
source-code/line-number data of a particular function by doing something
like this:

Get the line number of the beginning of n_tty_read(), which in my kernel
is at 1698 -- your's will probably be different:

crash> gdb list n_tty_read
1695 * This code must be sure never to sleep through a hangup.
1696 */
1697
1698 static ssize_t n_tty_read(struct tty_struct *tty, struct file *file,
1699 unsigned char __user *buf, size_t nr)
1700 {
1701 unsigned char __user *b = buf;
1702 DECLARE_WAITQUEUE(wait, current);
1703 int c;
1704 int minimum, time;
crash>

Then get the line number of the next function in the file, which is
n_tty_write():

crash> gdb list n_tty_write
1918 * lock themselves)
1919 */
1920
1921 static ssize_t n_tty_write(struct tty_struct *tty, struct file *file,
1922 const unsigned char *buf, size_t nr)
1923 {
1924 const unsigned char *b = buf;
1925 DECLARE_WAITQUEUE(wait, current);
1926 int c;
1927 ssize_t retval = 0;

And then dump the whole n_tty_read() function (plus some extra stuff):

crash> gdb list 1698,1920
1698 static ssize_t n_tty_read(struct tty_struct *tty, struct file *file,
1699 unsigned char __user *buf, size_t nr)
1700 {
1701 unsigned char __user *b = buf;
1702 DECLARE_WAITQUEUE(wait, current);
1703 int c;
1704 int minimum, time;
1705 ssize_t retval = 0;
1706 ssize_t size;
1707 long timeout;
1708 unsigned long flags;
1709 int packet;
1710
1711 do_it_again:
1712
1713 BUG_ON(!tty->read_buf);
1714
1715 c = job_control(tty, file);
1716 if (c < 0)
1717 return c;
1718
1719 minimum = time = 0;
1720 timeout = MAX_SCHEDULE_TIMEOUT;
1721 if (!tty->icanon) {
1722 time = (HZ / 10) * TIME_CHAR(tty);
1723 minimum = MIN_CHAR(tty);
...

And lastly, since the crash occurred at

IP: [<ffffffff811f03b3>] n_tty_read+0x58c/0x818

Do this:

crash> dis -rl n_tty_read+0x58c
...

And then post all of that data.

Dave

[Shashidhara Shamaiah]

Hi Dave,

Thank you so much for your help.

Below is the output of dis -rl n_tty_read+0x58c

crash> dis -rl n_tty_read+0x58c
dis: line numbers are not available
0xffffffff811efe27 <n_tty_read>: push %rbp
0xffffffff811efe28 <n_tty_read+1>: mov %gs:0xb500,%rax
0xffffffff811efe31 <n_tty_read+10>: mov %rsp,%rbp
0xffffffff811efe34 <n_tty_read+13>: push %r15
0xffffffff811efe36 <n_tty_read+15>: push %r14
0xffffffff811efe38 <n_tty_read+17>: push %r13
0xffffffff811efe3a <n_tty_read+19>: mov %rdi,%r13
0xffffffff811efe3d <n_tty_read+22>: lea -0x70(%rbp),%rdi
0xffffffff811efe41 <n_tty_read+26>: push %r12
0xffffffff811efe43 <n_tty_read+28>: push %rbx
0xffffffff811efe44 <n_tty_read+29>: lea 0x490(%r13),%rbx
0xffffffff811efe4b <n_tty_read+36>: sub $0xe8,%rsp
0xffffffff811efe52 <n_tty_read+43>: mov %rax,-0x98(%rbp)
0xffffffff811efe59 <n_tty_read+50>: mov %rcx,-0x78(%rbp)
0xffffffff811efe5d <n_tty_read+54>: xor %eax,%eax
0xffffffff811efe5f <n_tty_read+56>: mov $0xa,%ecx
0xffffffff811efe64 <n_tty_read+61>: mov %rdx,-0xd8(%rbp)
0xffffffff811efe6b <n_tty_read+68>: mov %rsi,-0xd0(%rbp)
0xffffffff811efe72 <n_tty_read+75>: mov %rdx,-0x40(%rbp)
0xffffffff811efe76 <n_tty_read+79>: rep stos %eax,%es:(%rdi)
0xffffffff811efe78 <n_tty_read+81>: lea 0x1c0(%r13),%rax
0xffffffff811efe7f <n_tty_read+88>: lea 0x1c8(%r13),%rcx
0xffffffff811efe86 <n_tty_read+95>: mov %rbx,-0xc0(%rbp)
0xffffffff811efe8d <n_tty_read+102>: lea 0xd8(%r13),%rbx
0xffffffff811efe94 <n_tty_read+109>: movq
$0xffffffff81045f84,-0x60(%rbp)
0xffffffff811efe9c <n_tty_read+117>: movq $0x0,-0xa8(%rbp)
0xffffffff811efea7 <n_tty_read+128>: mov -0x98(%rbp),%rdx
0xffffffff811efeae <n_tty_read+135>: mov %rax,-0xc8(%rbp)
0xffffffff811efeb5 <n_tty_read+142>: mov -0x98(%rbp),%rax
0xffffffff811efebc <n_tty_read+149>: mov %rcx,-0x90(%rbp)
0xffffffff811efec3 <n_tty_read+156>: lea 0x51c(%r13),%rcx
0xffffffff811efeca <n_tty_read+163>: mov %rbx,-0x80(%rbp)
0xffffffff811efece <n_tty_read+167>: mov %rdx,-0x68(%rbp)
0xffffffff811efed2 <n_tty_read+171>: lea 0x268(%r13),%rdx
0xffffffff811efed9 <n_tty_read+178>: mov %rcx,-0xb8(%rbp)
0xffffffff811efee0 <n_tty_read+185>: mov %rax,-0xf8(%rbp)
0xffffffff811efee7 <n_tty_read+192>: mov %rax,-0x100(%rbp)
0xffffffff811efeee <n_tty_read+199>: mov %rdx,-0x88(%rbp)
0xffffffff811efef5 <n_tty_read+206>: mov %rax,-0x108(%rbp)
0xffffffff811efefc <n_tty_read+213>: mov %rax,-0x110(%rbp)
0xffffffff811eff03 <n_tty_read+220>: cmpq $0x0,0x250(%r13)
0xffffffff811eff0b <n_tty_read+228>: jne 0xffffffff811eff11
<n_tty_read+234>
0xffffffff811eff0d <n_tty_read+230>: ud2a
0xffffffff811eff0f <n_tty_read+232>: jmp 0xffffffff811eff0f
<n_tty_read+232>
0xffffffff811eff11 <n_tty_read+234>: mov -0xd0(%rbp),%rdx
0xffffffff811eff18 <n_tty_read+241>: mov 0x20(%rdx),%rax
0xffffffff811eff1c <n_tty_read+245>: cmpq
$0xffffffff811ed61f,0x18(%rax)
0xffffffff811eff24 <n_tty_read+253>: je 0xffffffff811effef
<n_tty_read+456>
0xffffffff811eff2a <n_tty_read+259>: mov -0xf8(%rbp),%rcx
0xffffffff811eff31 <n_tty_read+266>: mov 0x478(%rcx),%rax
0xffffffff811eff38 <n_tty_read+273>: cmp %r13,0x180(%rax)
0xffffffff811eff3f <n_tty_read+280>: jne 0xffffffff811effef
<n_tty_read+456>
0xffffffff811eff45 <n_tty_read+286>: mov 0xc8(%r13),%rdx
0xffffffff811eff4c <n_tty_read+293>: test %rdx,%rdx
0xffffffff811eff4f <n_tty_read+296>: jne 0xffffffff811eff64
<n_tty_read+317>
0xffffffff811eff51 <n_tty_read+298>: mov $0xffffffff8139c972,%rdi
0xffffffff811eff58 <n_tty_read+305>: xor %eax,%eax
0xffffffff811eff5a <n_tty_read+307>: callq 0xffffffff812d4abf
<printk>
0xffffffff811eff5f <n_tty_read+312>: jmpq 0xffffffff811effef
<n_tty_read+456>
0xffffffff811eff64 <n_tty_read+317>: mov -0xf8(%rbp),%rbx
0xffffffff811eff6b <n_tty_read+324>: mov 0x1e0(%rbx),%rax
0xffffffff811eff72 <n_tty_read+331>: cmp %rdx,0x238(%rax)
0xffffffff811eff79 <n_tty_read+338>: je 0xffffffff811effef
<n_tty_read+456>
0xffffffff811eff7b <n_tty_read+340>: mov -0x98(%rbp),%rax
0xffffffff811eff82 <n_tty_read+347>: testb $0x10,0x48a(%rax)
0xffffffff811eff89 <n_tty_read+354>: jne 0xffffffff811f0611
<n_tty_read+2026>
0xffffffff811eff8f <n_tty_read+360>: mov 0x480(%rax),%rax
0xffffffff811eff96 <n_tty_read+367>: cmpq $0x1,0x288(%rax)
0xffffffff811eff9e <n_tty_read+375>: jne 0xffffffff811f0604
<n_tty_read+2013>
0xffffffff811effa4 <n_tty_read+381>: jmpq 0xffffffff811f0611
<n_tty_read+2026>
0xffffffff811effa9 <n_tty_read+386>: mov -0x98(%rbp),%rcx
0xffffffff811effb0 <n_tty_read+393>: mov $0x1,%edx
0xffffffff811effb5 <n_tty_read+398>: mov $0x15,%esi
0xffffffff811effba <n_tty_read+403>: mov 0x1e0(%rcx),%rax
0xffffffff811effc1 <n_tty_read+410>: mov 0x238(%rax),%rdi
0xffffffff811effc8 <n_tty_read+417>: callq 0xffffffff8105953a
<kill_pgrp>
0xffffffff811effcd <n_tty_read+422>: mov %gs:0xb508,%rdx
0xffffffff811effd6 <n_tty_read+431>: lea -0x1fc8(%rdx),%rax
0xffffffff811effdd <n_tty_read+438>: lock orb $0x4,-0x1fc8(%rdx)
0xffffffff811effe5 <n_tty_read+446>: mov $0xfffffe00,%eax
0xffffffff811effea <n_tty_read+451>: jmpq 0xffffffff811f0616
<n_tty_read+2031>
0xffffffff811effef <n_tty_read+456>: testb $0x10,0x21c(%r13)
0xffffffff811efff7 <n_tty_read+464>: je 0xffffffff811f000f
<n_tty_read+488>
0xffffffff811efff9 <n_tty_read+466>: movl $0x0,-0xb0(%rbp)
0xffffffff811f0003 <n_tty_read+476>: movl $0x0,-0xac(%rbp)
0xffffffff811f000d <n_tty_read+486>: jmp 0xffffffff811f0076
<n_tty_read+591>
0xffffffff811f000f <n_tty_read+488>: mov 0x70(%r13),%rdx
0xffffffff811f0013 <n_tty_read+492>: movzbl 0x16(%rdx),%eax
0xffffffff811f0017 <n_tty_read+496>: imul $0x19,%eax,%eax
0xffffffff811f001a <n_tty_read+499>: mov %eax,-0xac(%rbp)
0xffffffff811f0020 <n_tty_read+505>: movzbl 0x17(%rdx),%edx
0xffffffff811f0024 <n_tty_read+509>: test %edx,%edx
0xffffffff811f0026 <n_tty_read+511>: mov %edx,-0xb0(%rbp)
0xffffffff811f002c <n_tty_read+517>: je 0xffffffff811f0082
<n_tty_read+603>
0xffffffff811f002e <n_tty_read+519>: test %eax,%eax
0xffffffff811f0030 <n_tty_read+521>: je 0xffffffff811f003e
<n_tty_read+535>
0xffffffff811f0032 <n_tty_read+523>: movw $0x1,0x21e(%r13)
0xffffffff811f003c <n_tty_read+533>: jmp 0xffffffff811f0076
<n_tty_read+591>
0xffffffff811f003e <n_tty_read+535>: mov -0x90(%rbp),%rbx
0xffffffff811f0045 <n_tty_read+542>: cmp %rbx,0x1c8(%r13)
0xffffffff811f004c <n_tty_read+549>: je 0xffffffff811f0068
<n_tty_read+577>
0xffffffff811f004e <n_tty_read+551>: movzwl 0x21e(%r13),%eax
0xffffffff811f0056 <n_tty_read+559>: mov $0x7fffffffffffffff,%r15
0xffffffff811f0060 <n_tty_read+569>: cmp -0xb0(%rbp),%eax
0xffffffff811f0066 <n_tty_read+575>: jle 0xffffffff811f00b7
<n_tty_read+656>
0xffffffff811f0068 <n_tty_read+577>: mov -0xb0(%rbp),%eax
0xffffffff811f006e <n_tty_read+583>: mov %ax,0x21e(%r13)
0xffffffff811f0076 <n_tty_read+591>: mov $0x7fffffffffffffff,%r15
0xffffffff811f0080 <n_tty_read+601>: jmp 0xffffffff811f00b7
<n_tty_read+656>
0xffffffff811f0082 <n_tty_read+603>: movslq -0xac(%rbp),%r15
0xffffffff811f0089 <n_tty_read+610>: cmpl $0x0,-0xac(%rbp)
0xffffffff811f0090 <n_tty_read+617>: mov $0x0,%eax
0xffffffff811f0095 <n_tty_read+622>: movw $0x1,0x21e(%r13)
0xffffffff811f009f <n_tty_read+632>: movl $0x1,-0xb0(%rbp)
0xffffffff811f00a9 <n_tty_read+642>: movl $0x0,-0xac(%rbp)
0xffffffff811f00b3 <n_tty_read+652>: cmove %rax,%r15
0xffffffff811f00b7 <n_tty_read+656>: mov -0xd0(%rbp),%rdx
0xffffffff811f00be <n_tty_read+663>: testb $0x8,0x39(%rdx)
0xffffffff811f00c2 <n_tty_read+667>: je 0xffffffff811f00e4
<n_tty_read+701>
0xffffffff811f00c4 <n_tty_read+669>: mov -0xc0(%rbp),%rdi
0xffffffff811f00cb <n_tty_read+676>: callq 0xffffffff812d5ec7
<mutex_trylock>
0xffffffff811f00d0 <n_tty_read+681>: test %eax,%eax
0xffffffff811f00d2 <n_tty_read+683>: jne 0xffffffff811f0104
<n_tty_read+733>
0xffffffff811f00d4 <n_tty_read+685>: movq
$0xfffffffffffffff5,-0xa8(%rbp)
0xffffffff811f00df <n_tty_read+696>: jmpq 0xffffffff811f05eb
<n_tty_read+1988>
0xffffffff811f00e4 <n_tty_read+701>: mov -0xc0(%rbp),%rdi
0xffffffff811f00eb <n_tty_read+708>: callq 0xffffffff812d6358
<mutex_lock_interruptible>
0xffffffff811f00f0 <n_tty_read+713>: test %eax,%eax
0xffffffff811f00f2 <n_tty_read+715>: je 0xffffffff811f0104
<n_tty_read+733>
0xffffffff811f00f4 <n_tty_read+717>: movq
$0xfffffffffffffe00,-0xa8(%rbp)
0xffffffff811f00ff <n_tty_read+728>: jmpq 0xffffffff811f05eb
<n_tty_read+1988>
0xffffffff811f0104 <n_tty_read+733>: mov 0xec(%r13),%al
0xffffffff811f010b <n_tty_read+740>: mov -0xc8(%rbp),%rdi
0xffffffff811f0112 <n_tty_read+747>: lea -0x70(%rbp),%rsi
0xffffffff811f0116 <n_tty_read+751>: shr $0x3,%al
0xffffffff811f0119 <n_tty_read+754>: mov %eax,%ecx
0xffffffff811f011b <n_tty_read+756>: and $0x1,%ecx
0xffffffff811f011e <n_tty_read+759>: mov %ecx,-0x9c(%rbp)
0xffffffff811f0124 <n_tty_read+765>: callq 0xffffffff8106201b
<add_wait_queue>
0xffffffff811f0129 <n_tty_read+770>: movslq -0xb0(%rbp),%rbx
0xffffffff811f0130 <n_tty_read+777>: movslq -0xac(%rbp),%rax
0xffffffff811f0137 <n_tty_read+784>: mov -0xd8(%rbp),%rdx
0xffffffff811f013e <n_tty_read+791>: inc %rdx
0xffffffff811f0141 <n_tty_read+794>: mov %rbx,-0xe0(%rbp)
0xffffffff811f0148 <n_tty_read+801>: mov %rax,-0xe8(%rbp)
0xffffffff811f014f <n_tty_read+808>: mov %rdx,-0xf0(%rbp)
0xffffffff811f0156 <n_tty_read+815>: jmpq 0xffffffff811f0522
<n_tty_read+1787>
0xffffffff811f015b <n_tty_read+820>: cmpl $0x0,-0x9c(%rbp)
0xffffffff811f0162 <n_tty_read+827>: je 0xffffffff811f01ef
<n_tty_read+968>
0xffffffff811f0168 <n_tty_read+833>: mov 0xf8(%r13),%rax
0xffffffff811f016f <n_tty_read+840>: cmpb $0x0,0xed(%rax)
0xffffffff811f0176 <n_tty_read+847>: je 0xffffffff811f01ef
<n_tty_read+968>
0xffffffff811f0178 <n_tty_read+849>: mov -0xd8(%rbp),%rcx
0xffffffff811f017f <n_tty_read+856>: cmp %rcx,-0x40(%rbp)
0xffffffff811f0183 <n_tty_read+860>: jne 0xffffffff811f052d
<n_tty_read+1798>
0xffffffff811f0189 <n_tty_read+866>: lea 0x68(%rax),%rdi
0xffffffff811f018d <n_tty_read+870>: callq 0xffffffff812d6fb8
<_spin_lock_irqsave>
0xffffffff811f0192 <n_tty_read+875>: mov 0xf8(%r13),%rdi
0xffffffff811f0199 <n_tty_read+882>: mov %rax,%rsi
0xffffffff811f019c <n_tty_read+885>: mov 0xed(%rdi),%bl
0xffffffff811f01a2 <n_tty_read+891>: movb $0x0,0xed(%rdi)
0xffffffff811f01a9 <n_tty_read+898>: add $0x68,%rdi
0xffffffff811f01ad <n_tty_read+902>: callq 0xffffffff812d70c1
<_spin_unlock_irqrestore>
0xffffffff811f01b2 <n_tty_read+907>: mov -0x40(%rbp),%r12
0xffffffff811f01b6 <n_tty_read+911>: lea -0x31(%rbp),%rsi
0xffffffff811f01ba <n_tty_read+915>: mov $0x1,%edx
0xffffffff811f01bf <n_tty_read+920>: mov %r13,%rdi
0xffffffff811f01c2 <n_tty_read+923>: mov %bl,-0x31(%rbp)
0xffffffff811f01c5 <n_tty_read+926>: lea 0x1(%r12),%rax
0xffffffff811f01ca <n_tty_read+931>: mov %rax,-0x40(%rbp)
0xffffffff811f01ce <n_tty_read+935>: callq 0xffffffff812008ac
<tty_audit_add_data>
0xffffffff811f01d3 <n_tty_read+940>: mov -0x31(%rbp),%al
0xffffffff811f01d6 <n_tty_read+943>: mov %r12,%rcx
0xffffffff811f01d9 <n_tty_read+946>: callq 0xffffffff811949a0
<__put_user_1>
0xffffffff811f01de <n_tty_read+951>: test %eax,%eax
0xffffffff811f01e0 <n_tty_read+953>: jne 0xffffffff811f043d
<n_tty_read+1558>
0xffffffff811f01e6 <n_tty_read+959>: decq -0x78(%rbp)
0xffffffff811f01ea <n_tty_read+963>: jmpq 0xffffffff811f052d
<n_tty_read+1798>
0xffffffff811f01ef <n_tty_read+968>: mov $0x1,%eax
0xffffffff811f01f4 <n_tty_read+973>: mov -0x100(%rbp),%rbx
0xffffffff811f01fb <n_tty_read+980>: xchg %rax,(%rbx)
0xffffffff811f01fe <n_tty_read+983>: mov -0x40(%rbp),%rcx
0xffffffff811f0202 <n_tty_read+987>: mov -0xd8(%rbp),%rax
0xffffffff811f0209 <n_tty_read+994>: mov -0xe0(%rbp),%rbx
0xffffffff811f0210 <n_tty_read+1001>: sub %rcx,%rax
0xffffffff811f0213 <n_tty_read+1004>: lea (%rax,%rbx,1),%rdx
0xffffffff811f0217 <n_tty_read+1008>: movzwl 0x21e(%r13),%eax
0xffffffff811f021f <n_tty_read+1016>: cmp %rax,%rdx
0xffffffff811f0222 <n_tty_read+1019>: jge 0xffffffff811f0240
<n_tty_read+1049>
0xffffffff811f0224 <n_tty_read+1021>: test %rdx,%rdx
0xffffffff811f0227 <n_tty_read+1024>: jle 0xffffffff811f0240
<n_tty_read+1049>
0xffffffff811f0229 <n_tty_read+1026>: mov -0xd8(%rbp),%eax
0xffffffff811f022f <n_tty_read+1032>: sub %cx,%ax
0xffffffff811f0232 <n_tty_read+1035>: add -0xb0(%rbp),%eax
0xffffffff811f0238 <n_tty_read+1041>: mov %ax,0x21e(%r13)
0xffffffff811f0240 <n_tty_read+1049>: mov %r13,%rdi
0xffffffff811f0243 <n_tty_read+1052>: callq 0xffffffff811f37f3
<tty_flush_to_ldisc>
0xffffffff811f0248 <n_tty_read+1057>: testb $0x10,0x21c(%r13)
0xffffffff811f0250 <n_tty_read+1065>: je 0xffffffff811f0261
<n_tty_read+1082>
0xffffffff811f0252 <n_tty_read+1067>: cmpl $0x0,0x478(%r13)
0xffffffff811f025a <n_tty_read+1075>: jne 0xffffffff811f026f
<n_tty_read+1096>
0xffffffff811f025c <n_tty_read+1077>: jmpq 0xffffffff811f0621
<n_tty_read+2042>
0xffffffff811f0261 <n_tty_read+1082>: cmpl $0x0,0x260(%r13)
0xffffffff811f0269 <n_tty_read+1090>: jle 0xffffffff811f0621
<n_tty_read+2042>
0xffffffff811f026f <n_tty_read+1096>: mov -0x110(%rbp),%rax
0xffffffff811f0276 <n_tty_read+1103>: movq $0x0,(%rax)
0xffffffff811f027d <n_tty_read+1110>: cmpl $0x0,-0x9c(%rbp)
0xffffffff811f0284 <n_tty_read+1117>: mov -0x40(%rbp),%rax
0xffffffff811f0288 <n_tty_read+1121>: je 0xffffffff811f0376
<n_tty_read+1359>
0xffffffff811f028e <n_tty_read+1127>: cmp -0xd8(%rbp),%rax
0xffffffff811f0295 <n_tty_read+1134>: jne 0xffffffff811f0376
<n_tty_read+1359>
0xffffffff811f029b <n_tty_read+1140>: jmpq 0xffffffff811f033b
<n_tty_read+1300>
0xffffffff811f02a0 <n_tty_read+1145>: mov -0xd0(%rbp),%rdi
0xffffffff811f02a7 <n_tty_read+1152>: callq 0xffffffff811eb980
<tty_hung_up_p>
0xffffffff811f02ac <n_tty_read+1157>: test %eax,%eax
0xffffffff811f02ae <n_tty_read+1159>: jne 0xffffffff811f052d
<n_tty_read+1798>
0xffffffff811f02b4 <n_tty_read+1165>: test %r15,%r15
0xffffffff811f02b7 <n_tty_read+1168>: je 0xffffffff811f052d
<n_tty_read+1798>
0xffffffff811f02bd <n_tty_read+1174>: mov -0xd0(%rbp),%rdx
0xffffffff811f02c4 <n_tty_read+1181>: testb $0x8,0x39(%rdx)
0xffffffff811f02c8 <n_tty_read+1185>: je 0xffffffff811f02da
<n_tty_read+1203>
0xffffffff811f02ca <n_tty_read+1187>: movq
$0xfffffffffffffff5,-0xa8(%rbp)
0xffffffff811f02d5 <n_tty_read+1198>: jmpq 0xffffffff811f052d
<n_tty_read+1798>
0xffffffff811f02da <n_tty_read+1203>: mov -0x108(%rbp),%rcx
0xffffffff811f02e1 <n_tty_read+1210>: mov 0x8(%rcx),%rax
0xffffffff811f02e5 <n_tty_read+1214>: testb $0x4,0x10(%rax)
0xffffffff811f02e9 <n_tty_read+1218>: je 0xffffffff811f02fb
<n_tty_read+1236>
0xffffffff811f02eb <n_tty_read+1220>: movq
$0xfffffffffffffe00,-0xa8(%rbp)
0xffffffff811f02f6 <n_tty_read+1231>: jmpq 0xffffffff811f052d
<n_tty_read+1798>
0xffffffff811f02fb <n_tty_read+1236>: mov $0xfff,%eax
0xffffffff811f0300 <n_tty_read+1241>: sub 0x260(%r13),%eax
0xffffffff811f0307 <n_tty_read+1248>: test %eax,%eax
0xffffffff811f0309 <n_tty_read+1250>: jg 0xffffffff811f0324
<n_tty_read+1277>
0xffffffff811f030b <n_tty_read+1252>: xor %eax,%eax
0xffffffff811f030d <n_tty_read+1254>: testb $0x10,0x21c(%r13)
0xffffffff811f0315 <n_tty_read+1262>: je 0xffffffff811f0324
<n_tty_read+1277>
0xffffffff811f0317 <n_tty_read+1264>: xor %eax,%eax
0xffffffff811f0319 <n_tty_read+1266>: cmpl $0x0,0x478(%r13)
0xffffffff811f0321 <n_tty_read+1274>: sete %al
0xffffffff811f0324 <n_tty_read+1277>: mov %r15,%rdi
0xffffffff811f0327 <n_tty_read+1280>: mov %eax,0xf0(%r13)
0xffffffff811f032e <n_tty_read+1287>: callq 0xffffffff812d5a02
<schedule_timeout>
0xffffffff811f0333 <n_tty_read+1292>: mov %rax,%r15
0xffffffff811f0336 <n_tty_read+1295>: jmpq 0xffffffff811f0522
<n_tty_read+1787>
0xffffffff811f033b <n_tty_read+1300>: mov -0xf0(%rbp),%rbx
0xffffffff811f0342 <n_tty_read+1307>: lea -0x31(%rbp),%rsi
0xffffffff811f0346 <n_tty_read+1311>: mov $0x1,%edx
0xffffffff811f034b <n_tty_read+1316>: mov %r13,%rdi
0xffffffff811f034e <n_tty_read+1319>: movb $0x0,-0x31(%rbp)
0xffffffff811f0352 <n_tty_read+1323>: mov %rbx,-0x40(%rbp)
0xffffffff811f0356 <n_tty_read+1327>: callq 0xffffffff812008ac
<tty_audit_add_data>
0xffffffff811f035b <n_tty_read+1332>: mov -0x31(%rbp),%al
0xffffffff811f035e <n_tty_read+1335>: mov -0xd8(%rbp),%rcx
0xffffffff811f0365 <n_tty_read+1342>: callq 0xffffffff811949a0
<__put_user_1>
0xffffffff811f036a <n_tty_read+1347>: test %eax,%eax
0xffffffff811f036c <n_tty_read+1349>: jne 0xffffffff811f043d
<n_tty_read+1558>
0xffffffff811f0372 <n_tty_read+1355>: decq -0x78(%rbp)
0xffffffff811f0376 <n_tty_read+1359>: testb $0x10,0x21c(%r13)
0xffffffff811f037e <n_tty_read+1367>: jne 0xffffffff811f0456
<n_tty_read+1583>
0xffffffff811f0384 <n_tty_read+1373>: jmpq 0xffffffff811f047a
<n_tty_read+1619>
0xffffffff811f0389 <n_tty_read+1378>: mov 0x25c(%r13),%eax
0xffffffff811f0390 <n_tty_read+1385>: mov -0x88(%rbp),%rbx
0xffffffff811f0397 <n_tty_read+1392>: lock btr %eax,(%rbx)
0xffffffff811f039b <n_tty_read+1396>: sbb %r14d,%r14d
0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx
0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax
0xffffffff811f03ac <n_tty_read+1413>: mov -0xb8(%rbp),%rdi
0xffffffff811f03b3 <n_tty_read+1420>: movsbl (%rax,%rdx,1),%ebx

Below is the output of bt -a command in crash

bt -a
PID: 0 TASK: ffffffff814204b0 CPU: 0 COMMAND: "swapper"
#0 [ffff880033007e80] crash_nmi_callback at ffffffff8101fbc9
#1 [ffff880033007e90] notifier_call_chain at ffffffff81065893
#2 [ffff880033007ed0] atomic_notifier_call_chain at ffffffff810658dd
#3 [ffff880033007ee0] notify_die at ffffffff8106597f
#4 [ffff880033007f10] do_nmi at ffffffff8100dc5d
#5 [ffff880033007f50] nmi at ffffffff812d76b0
[exception RIP: mwait_idle+163]
RIP: ffffffff81013029 RSP: ffffffff813e3eb8 RFLAGS: 00000246
RAX: 0000000000000000 RBX: ffffffff813e3fd8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffffffff813e3fd8 RDI: ffffffff81522308
RBP: ffffffff813e3ec8 R8: 0000000000000000 R9: ffff88003306e290
R10: 0000000000012d80 R11: 0000000000000000 R12: ffffffff8147e368
R13: ffffffff814ccb30 R14: ffffffff814cdfa0 R15: ffffffff813e3fa8
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
--- <NMI exception stack> ---
#6 [ffffffff813e3eb8] mwait_idle at ffffffff81013029
#7 [ffffffff813e3ed0] cpu_idle at ffffffff8100af21

PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"
#0 [ffff88031ce759d0] machine_kexec at ffffffff81024486
#1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230
#2 [ffff88031ce75b20] oops_end at ffffffff8100fa38
#3 [ffff88031ce75b50] no_context at ffffffff8102d801
#4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9
#5 [ffff88031ce75c70] bad_area at ffffffff8102da41
#6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19
#7 [ffff88031ce75cf0] page_fault at ffffffff812d7425
[exception RIP: n_tty_read+1420]
RIP: ffffffff811f03b3 RSP: ffff88031ce75da8 RFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044
RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c
RBP: ffff88031ce75eb8 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044
R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#8 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e
#9 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8
#10 [ffff88031ce75f40] sys_read at ffffffff810ebe48
#11 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2
RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212
RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000
RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000
RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b

PID: 0 TASK: ffff88031e0e3540 CPU: 2 COMMAND: "swapper"
#0 [ffff880033047e80] crash_nmi_callback at ffffffff8101fbc9
#1 [ffff880033047e90] notifier_call_chain at ffffffff81065893
#2 [ffff880033047ed0] atomic_notifier_call_chain at ffffffff810658dd
#3 [ffff880033047ee0] notify_die at ffffffff8106597f
#4 [ffff880033047f10] do_nmi at ffffffff8100dc5d
#5 [ffff880033047f50] nmi at ffffffff812d76b0
[exception RIP: mwait_idle+163]
RIP: ffffffff81013029 RSP: ffff88031e0e5ef8 RFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff88031e0e5fd8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88031e0e5fd8 RDI: ffffffff81522308
RBP: ffff88031e0e5f08 R8: 0000000000000000 R9: ffff88003302e290
R10: 0000000000012d80 R11: 0000000000000000 R12: ffffffff8147e368
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
--- <NMI exception stack> ---
#6 [ffff88031e0e5ef8] mwait_idle at ffffffff81013029
#7 [ffff88031e0e5f10] cpu_idle at ffffffff8100af21

PID: 0 TASK: ffff88031e113580 CPU: 3 COMMAND: "swapper"
#0 [ffff880033067e80] crash_nmi_callback at ffffffff8101fbc9
#1 [ffff880033067e90] notifier_call_chain at ffffffff81065893
#2 [ffff880033067ed0] atomic_notifier_call_chain at ffffffff810658dd
#3 [ffff880033067ee0] notify_die at ffffffff8106597f
#4 [ffff880033067f10] do_nmi at ffffffff8100dc5d
#5 [ffff880033067f50] nmi at ffffffff812d76b0
[exception RIP: mwait_idle+163]
RIP: ffffffff81013029 RSP: ffff88031e115ef8 RFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff88031e115fd8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88031e115fd8 RDI: ffffffff81522308
RBP: ffff88031e115f08 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000800 R11: 0000000000000000 R12: ffffffff8147e368
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
--- <NMI exception stack> ---
#6 [ffff88031e115ef8] mwait_idle at ffffffff81013029
#7 [ffff88031e115f10] cpu_idle at ffffffff8100af21

Please let me know if you need any other details.

Thanks and Regards
Shashidhara


-----Original Message-----
From: crash-utility-***@redhat.com
[mailto:crash-utility-***@redhat.com] On Behalf Of Dave Anderson
Sent: Thursday, June 23, 2011 9:35 PM
To: Discussion list for crash utility usage,maintenance and development
Subject: Re: [Crash-utility] Unable to switch stack frames while using
crash



----- Original Message -----

Post by Dave Anderson
BTW, are you sure about that?
Presuming that the "tty" pointer is ffff8802cbd54800 as you've shown below,
and therefore tty->read_buf is 0xffff8802cbfe6000 and tty->read_tail is 0,
then the statement above would be simply be reading tty->read_buf[0], or
virtual address 0xffff8802cbfe6000. But the oops shows it faulting on a
BUG: unable to handle kernel NULL pointer dereference at

0000000000000005

Just for my own sanity, can you either attach the "drivers/char/n_tty.c"

from *your* specific kernel, or get the source-code/line-number data
from
the embedded gdb module?

If you don't have the n_tty.c file readily available, you can get the
source-code/line-number data of a particular function by doing something
like this:

Get the line number of the beginning of n_tty_read(), which in my kernel
is at 1698 -- your's will probably be different:

crash> gdb list n_tty_read
1695 * This code must be sure never to sleep through a hangup.
1696 */
1697
1698 static ssize_t n_tty_read(struct tty_struct *tty, struct file
*file,
1699 unsigned char __user *buf, size_t nr)
1700 {
1701 unsigned char __user *b = buf;
1702 DECLARE_WAITQUEUE(wait, current);
1703 int c;
1704 int minimum, time;
crash>

Then get the line number of the next function in the file, which is
n_tty_write():

crash> gdb list n_tty_write
1918 * lock themselves)
1919 */
1920
1921 static ssize_t n_tty_write(struct tty_struct *tty, struct file
*file,
1922 const unsigned char *buf, size_t nr)
1923 {
1924 const unsigned char *b = buf;
1925 DECLARE_WAITQUEUE(wait, current);
1926 int c;
1927 ssize_t retval = 0;

And then dump the whole n_tty_read() function (plus some extra stuff):

crash> gdb list 1698,1920
1698 static ssize_t n_tty_read(struct tty_struct *tty, struct file
*file,
1699 unsigned char __user *buf, size_t nr)
1700 {
1701 unsigned char __user *b = buf;
1702 DECLARE_WAITQUEUE(wait, current);
1703 int c;
1704 int minimum, time;
1705 ssize_t retval = 0;
1706 ssize_t size;
1707 long timeout;
1708 unsigned long flags;
1709 int packet;
1710
1711 do_it_again:
1712
1713 BUG_ON(!tty->read_buf);
1714
1715 c = job_control(tty, file);
1716 if (c < 0)
1717 return c;
1718
1719 minimum = time = 0;
1720 timeout = MAX_SCHEDULE_TIMEOUT;
1721 if (!tty->icanon) {
1722 time = (HZ / 10) * TIME_CHAR(tty);
1723 minimum = MIN_CHAR(tty);
...

And lastly, since the crash occurred at

IP: [<ffffffff811f03b3>] n_tty_read+0x58c/0x818

Do this:

crash> dis -rl n_tty_read+0x58c
...

And then post all of that data.

Dave


--
Crash-utility mailing list
Crash-***@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Information transmitted by this e-mail is proprietary to MphasiS, its associated companies and/ or its customers and is intended
for use only by the individual or entity to which it is addressed, and may contain information that is privileged, confidential or
exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded
to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly
prohibited. In such cases, please notify us immediately at ***@mphasis.com and delete this mail from your records.

[Dave Anderson]

----- Original Message -----

Post by Dave Anderson
If you don't have the n_tty.c file readily available, you can get the
source-code/line-number data of a particular function by doing something

Actually, this is a much easier way:

crash> gdb list n_tty_read,n_tty_write

Dave

[Dave Anderson]

---- Original Message -----

Post by Dave Anderson

Post by Dave Anderson
BTW, are you sure about that?
Presuming that the "tty" pointer is ffff8802cbd54800 as you've shown below,
and therefore tty->read_buf is 0xffff8802cbfe6000 and tty->read_tail is 0,
then the statement above would be simply be reading tty->read_buf[0], or
virtual address 0xffff8802cbfe6000. But the oops shows it faulting on a

Well, as it turns out, you have every reason to be sure about that...

Anyway, I don't understand why line numbers are not available with

Post by Dave Anderson
crash> dis -rl n_tty_read+0x58c
dis: line numbers are not available

... [ cut ] ...

Post by Dave Anderson
0xffffffff811f039b <n_tty_read+1396>: sbb %r14d,%r14d
0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx
0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax
0xffffffff811f03ac <n_tty_read+1413>: mov -0xb8(%rbp),%rdi
0xffffffff811f03b3 <n_tty_read+1420>: movsbl (%rax,%rdx,1),%ebx

But nonetheless, there is only on movsbl instruction in n_tty_read(),
and looking at a RHEL6 kernel, you were correct in your original
determination of the faulting instruction:

crash> dis n_tty_read | grep movsbl
0xffffffff812f88c9 <n_tty_read+0x2c9>: movsbl (%rdx,%rax,1),%ebx
crash> dis -rl 0xffffffff812f88c9 | tail
... [ cut ] ...
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/drivers/char/n_tty.c: 1821
0xffffffff812f88c9 <n_tty_read+0x2c9>: movsbl (%rdx,%rax,1),%ebx
crash>

which is here:

1814 if (tty->icanon) {
1815 /* N.B. avoid overrun if nr == 0 */
1816 while (nr && tty->read_cnt) {
1817 int eol;
1818
1819 eol = test_and_clear_bit(tty->read_tail,
1820 tty->read_flags);
1821 c = tty->read_buf[tty->read_tail];

The tty_struct offsets are these:

crash> tty_struct -o
struct tty_struct {
... [ cut ]...
[0x250] char *read_buf;
[0x258] int read_head;
[0x25c] int read_tail;
...

And you can see in the previous instructions the tty->read_buf (0x250)
and tty->read_tail (0x25c) offsets being added to the tty_struct

Post by Dave Anderson
0xffffffff811f039b <n_tty_read+1396>: sbb %r14d,%r14d
0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx
0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax
0xffffffff811f03ac <n_tty_read+1413>: mov -0xb8(%rbp),%rdi
0xffffffff811f03b3 <n_tty_read+1420>: movsbl (%rax,%rdx,1),%ebx

And as you originally reported, the tty_struct address in R13
is ffff8802cbd54800:

PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"
#0 [ffff88031ce759d0] machine_kexec at ffffffff81024486
#1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230
#2 [ffff88031ce75b20] oops_end at ffffffff8100fa38
#3 [ffff88031ce75b50] no_context at ffffffff8102d801
#4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9
#5 [ffff88031ce75c70] bad_area at ffffffff8102da41
#6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19
#7 [ffff88031ce75cf0] page_fault at ffffffff812d7425
[exception RIP: n_tty_read+1420]
RIP: ffffffff811f03b3 RSP: ffff88031ce75da8 RFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044
RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c
RBP: ffff88031ce75eb8 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044
R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#8 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e
#9 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8
#10 [ffff88031ce75f40] sys_read at ffffffff810ebe48
#11 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2
RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212
RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000
RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000
RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b

But for whatever reason -- and I cannot explain it -- after these

Post by Dave Anderson
0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx
0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax

the RDX register ended up with 0000000000000005, and the RAX register with
a 0000000000000000, leading to the:

BUG: unable to handle kernel NULL pointer dereference at 0000000000000005

But when you display the tty_struct at ffff8802cbd54800, you see the
read_buf and read_tail with seemingly legitimate values:

crash> tty_struct ffff8802cbd54800
struct tty_struct { ...
magic = 21505,
driver = 0xffff88031b54ea00,
ops = 0xffffffff8130f650,
name = "pts9\000\...",
driver_data = 0xffff88029c8a9668,
icanon = 1 '\001',
read_buf = 0xffff8802cbfe6000 "",
read_head = 0,
read_tail = 0,
read_cnt = 0,
...

So everything in your analysis was correct, but how it is possible
that the RDX and RAX registers to have ended up with 0 and 5 is hard
to explain. And for that matter, since tty->read_cnt is 0 above,
your original question as to how that code path was taken to
begin with is also valid.

So I don't know -- anybody on the list ever seen anything like this?

Stumped,
Dave


----- Original Message -----

Post by Dave Anderson
Hi Dave,
Thank you so much for your help.
Below is the output of dis -rl n_tty_read+0x58c
crash> dis -rl n_tty_read+0x58c
dis: line numbers are not available
0xffffffff811efe27 <n_tty_read>: push %rbp
0xffffffff811efe28 <n_tty_read+1>: mov %gs:0xb500,%rax
0xffffffff811efe31 <n_tty_read+10>: mov %rsp,%rbp
0xffffffff811efe34 <n_tty_read+13>: push %r15
0xffffffff811efe36 <n_tty_read+15>: push %r14
0xffffffff811efe38 <n_tty_read+17>: push %r13
0xffffffff811efe3a <n_tty_read+19>: mov %rdi,%r13
0xffffffff811efe3d <n_tty_read+22>: lea -0x70(%rbp),%rdi
0xffffffff811efe41 <n_tty_read+26>: push %r12
0xffffffff811efe43 <n_tty_read+28>: push %rbx
0xffffffff811efe44 <n_tty_read+29>: lea 0x490(%r13),%rbx
0xffffffff811efe4b <n_tty_read+36>: sub $0xe8,%rsp
0xffffffff811efe52 <n_tty_read+43>: mov %rax,-0x98(%rbp)
0xffffffff811efe59 <n_tty_read+50>: mov %rcx,-0x78(%rbp)
0xffffffff811efe5d <n_tty_read+54>: xor %eax,%eax
0xffffffff811efe5f <n_tty_read+56>: mov $0xa,%ecx
0xffffffff811efe64 <n_tty_read+61>: mov %rdx,-0xd8(%rbp)
0xffffffff811efe6b <n_tty_read+68>: mov %rsi,-0xd0(%rbp)
0xffffffff811efe72 <n_tty_read+75>: mov %rdx,-0x40(%rbp)
0xffffffff811efe76 <n_tty_read+79>: rep stos %eax,%es:(%rdi)
0xffffffff811efe78 <n_tty_read+81>: lea 0x1c0(%r13),%rax
0xffffffff811efe7f <n_tty_read+88>: lea 0x1c8(%r13),%rcx
0xffffffff811efe86 <n_tty_read+95>: mov %rbx,-0xc0(%rbp)
0xffffffff811efe8d <n_tty_read+102>: lea 0xd8(%r13),%rbx
0xffffffff811efe94 <n_tty_read+109>: movq
$0xffffffff81045f84,-0x60(%rbp)
0xffffffff811efe9c <n_tty_read+117>: movq $0x0,-0xa8(%rbp)
0xffffffff811efea7 <n_tty_read+128>: mov -0x98(%rbp),%rdx
0xffffffff811efeae <n_tty_read+135>: mov %rax,-0xc8(%rbp)
0xffffffff811efeb5 <n_tty_read+142>: mov -0x98(%rbp),%rax
0xffffffff811efebc <n_tty_read+149>: mov %rcx,-0x90(%rbp)
0xffffffff811efec3 <n_tty_read+156>: lea 0x51c(%r13),%rcx
0xffffffff811efeca <n_tty_read+163>: mov %rbx,-0x80(%rbp)
0xffffffff811efece <n_tty_read+167>: mov %rdx,-0x68(%rbp)
0xffffffff811efed2 <n_tty_read+171>: lea 0x268(%r13),%rdx
0xffffffff811efed9 <n_tty_read+178>: mov %rcx,-0xb8(%rbp)
0xffffffff811efee0 <n_tty_read+185>: mov %rax,-0xf8(%rbp)
0xffffffff811efee7 <n_tty_read+192>: mov %rax,-0x100(%rbp)
0xffffffff811efeee <n_tty_read+199>: mov %rdx,-0x88(%rbp)
0xffffffff811efef5 <n_tty_read+206>: mov %rax,-0x108(%rbp)
0xffffffff811efefc <n_tty_read+213>: mov %rax,-0x110(%rbp)
0xffffffff811eff03 <n_tty_read+220>: cmpq $0x0,0x250(%r13)
0xffffffff811eff0b <n_tty_read+228>: jne 0xffffffff811eff11
<n_tty_read+234>
0xffffffff811eff0d <n_tty_read+230>: ud2a
0xffffffff811eff0f <n_tty_read+232>: jmp 0xffffffff811eff0f
<n_tty_read+232>
0xffffffff811eff11 <n_tty_read+234>: mov -0xd0(%rbp),%rdx
0xffffffff811eff18 <n_tty_read+241>: mov 0x20(%rdx),%rax
0xffffffff811eff1c <n_tty_read+245>: cmpq
$0xffffffff811ed61f,0x18(%rax)
0xffffffff811eff24 <n_tty_read+253>: je 0xffffffff811effef
<n_tty_read+456>
0xffffffff811eff2a <n_tty_read+259>: mov -0xf8(%rbp),%rcx
0xffffffff811eff31 <n_tty_read+266>: mov 0x478(%rcx),%rax
0xffffffff811eff38 <n_tty_read+273>: cmp %r13,0x180(%rax)
0xffffffff811eff3f <n_tty_read+280>: jne 0xffffffff811effef
<n_tty_read+456>
0xffffffff811eff45 <n_tty_read+286>: mov 0xc8(%r13),%rdx
0xffffffff811eff4c <n_tty_read+293>: test %rdx,%rdx
0xffffffff811eff4f <n_tty_read+296>: jne 0xffffffff811eff64
<n_tty_read+317>
0xffffffff811eff51 <n_tty_read+298>: mov $0xffffffff8139c972,%rdi
0xffffffff811eff58 <n_tty_read+305>: xor %eax,%eax
0xffffffff811eff5a <n_tty_read+307>: callq 0xffffffff812d4abf
<printk>
0xffffffff811eff5f <n_tty_read+312>: jmpq 0xffffffff811effef
<n_tty_read+456>
0xffffffff811eff64 <n_tty_read+317>: mov -0xf8(%rbp),%rbx
0xffffffff811eff6b <n_tty_read+324>: mov 0x1e0(%rbx),%rax
0xffffffff811eff72 <n_tty_read+331>: cmp %rdx,0x238(%rax)
0xffffffff811eff79 <n_tty_read+338>: je 0xffffffff811effef
<n_tty_read+456>
0xffffffff811eff7b <n_tty_read+340>: mov -0x98(%rbp),%rax
0xffffffff811eff82 <n_tty_read+347>: testb $0x10,0x48a(%rax)
0xffffffff811eff89 <n_tty_read+354>: jne 0xffffffff811f0611
<n_tty_read+2026>
0xffffffff811eff8f <n_tty_read+360>: mov 0x480(%rax),%rax
0xffffffff811eff96 <n_tty_read+367>: cmpq $0x1,0x288(%rax)
0xffffffff811eff9e <n_tty_read+375>: jne 0xffffffff811f0604
<n_tty_read+2013>
0xffffffff811effa4 <n_tty_read+381>: jmpq 0xffffffff811f0611
<n_tty_read+2026>
0xffffffff811effa9 <n_tty_read+386>: mov -0x98(%rbp),%rcx
0xffffffff811effb0 <n_tty_read+393>: mov $0x1,%edx
0xffffffff811effb5 <n_tty_read+398>: mov $0x15,%esi
0xffffffff811effba <n_tty_read+403>: mov 0x1e0(%rcx),%rax
0xffffffff811effc1 <n_tty_read+410>: mov 0x238(%rax),%rdi
0xffffffff811effc8 <n_tty_read+417>: callq 0xffffffff8105953a
<kill_pgrp>
0xffffffff811effcd <n_tty_read+422>: mov %gs:0xb508,%rdx
0xffffffff811effd6 <n_tty_read+431>: lea -0x1fc8(%rdx),%rax
0xffffffff811effdd <n_tty_read+438>: lock orb $0x4,-0x1fc8(%rdx)
0xffffffff811effe5 <n_tty_read+446>: mov $0xfffffe00,%eax
0xffffffff811effea <n_tty_read+451>: jmpq 0xffffffff811f0616
<n_tty_read+2031>
0xffffffff811effef <n_tty_read+456>: testb $0x10,0x21c(%r13)
0xffffffff811efff7 <n_tty_read+464>: je 0xffffffff811f000f
<n_tty_read+488>
0xffffffff811efff9 <n_tty_read+466>: movl $0x0,-0xb0(%rbp)
0xffffffff811f0003 <n_tty_read+476>: movl $0x0,-0xac(%rbp)
0xffffffff811f000d <n_tty_read+486>: jmp 0xffffffff811f0076
<n_tty_read+591>
0xffffffff811f000f <n_tty_read+488>: mov 0x70(%r13),%rdx
0xffffffff811f0013 <n_tty_read+492>: movzbl 0x16(%rdx),%eax
0xffffffff811f0017 <n_tty_read+496>: imul $0x19,%eax,%eax
0xffffffff811f001a <n_tty_read+499>: mov %eax,-0xac(%rbp)
0xffffffff811f0020 <n_tty_read+505>: movzbl 0x17(%rdx),%edx
0xffffffff811f0024 <n_tty_read+509>: test %edx,%edx
0xffffffff811f0026 <n_tty_read+511>: mov %edx,-0xb0(%rbp)
0xffffffff811f002c <n_tty_read+517>: je 0xffffffff811f0082
<n_tty_read+603>
0xffffffff811f002e <n_tty_read+519>: test %eax,%eax
0xffffffff811f0030 <n_tty_read+521>: je 0xffffffff811f003e
<n_tty_read+535>
0xffffffff811f0032 <n_tty_read+523>: movw $0x1,0x21e(%r13)
0xffffffff811f003c <n_tty_read+533>: jmp 0xffffffff811f0076
<n_tty_read+591>
0xffffffff811f003e <n_tty_read+535>: mov -0x90(%rbp),%rbx
0xffffffff811f0045 <n_tty_read+542>: cmp %rbx,0x1c8(%r13)
0xffffffff811f004c <n_tty_read+549>: je 0xffffffff811f0068
<n_tty_read+577>
0xffffffff811f004e <n_tty_read+551>: movzwl 0x21e(%r13),%eax
0xffffffff811f0056 <n_tty_read+559>: mov $0x7fffffffffffffff,%r15
0xffffffff811f0060 <n_tty_read+569>: cmp -0xb0(%rbp),%eax
0xffffffff811f0066 <n_tty_read+575>: jle 0xffffffff811f00b7
<n_tty_read+656>
0xffffffff811f0068 <n_tty_read+577>: mov -0xb0(%rbp),%eax
0xffffffff811f006e <n_tty_read+583>: mov %ax,0x21e(%r13)
0xffffffff811f0076 <n_tty_read+591>: mov $0x7fffffffffffffff,%r15
0xffffffff811f0080 <n_tty_read+601>: jmp 0xffffffff811f00b7
<n_tty_read+656>
0xffffffff811f0082 <n_tty_read+603>: movslq -0xac(%rbp),%r15
0xffffffff811f0089 <n_tty_read+610>: cmpl $0x0,-0xac(%rbp)
0xffffffff811f0090 <n_tty_read+617>: mov $0x0,%eax
0xffffffff811f0095 <n_tty_read+622>: movw $0x1,0x21e(%r13)
0xffffffff811f009f <n_tty_read+632>: movl $0x1,-0xb0(%rbp)
0xffffffff811f00a9 <n_tty_read+642>: movl $0x0,-0xac(%rbp)
0xffffffff811f00b3 <n_tty_read+652>: cmove %rax,%r15
0xffffffff811f00b7 <n_tty_read+656>: mov -0xd0(%rbp),%rdx
0xffffffff811f00be <n_tty_read+663>: testb $0x8,0x39(%rdx)
0xffffffff811f00c2 <n_tty_read+667>: je 0xffffffff811f00e4
<n_tty_read+701>
0xffffffff811f00c4 <n_tty_read+669>: mov -0xc0(%rbp),%rdi
0xffffffff811f00cb <n_tty_read+676>: callq 0xffffffff812d5ec7
<mutex_trylock>
0xffffffff811f00d0 <n_tty_read+681>: test %eax,%eax
0xffffffff811f00d2 <n_tty_read+683>: jne 0xffffffff811f0104
<n_tty_read+733>
0xffffffff811f00d4 <n_tty_read+685>: movq
$0xfffffffffffffff5,-0xa8(%rbp)
0xffffffff811f00df <n_tty_read+696>: jmpq 0xffffffff811f05eb
<n_tty_read+1988>
0xffffffff811f00e4 <n_tty_read+701>: mov -0xc0(%rbp),%rdi
0xffffffff811f00eb <n_tty_read+708>: callq 0xffffffff812d6358
<mutex_lock_interruptible>
0xffffffff811f00f0 <n_tty_read+713>: test %eax,%eax
0xffffffff811f00f2 <n_tty_read+715>: je 0xffffffff811f0104
<n_tty_read+733>
0xffffffff811f00f4 <n_tty_read+717>: movq
$0xfffffffffffffe00,-0xa8(%rbp)
0xffffffff811f00ff <n_tty_read+728>: jmpq 0xffffffff811f05eb
<n_tty_read+1988>
0xffffffff811f0104 <n_tty_read+733>: mov 0xec(%r13),%al
0xffffffff811f010b <n_tty_read+740>: mov -0xc8(%rbp),%rdi
0xffffffff811f0112 <n_tty_read+747>: lea -0x70(%rbp),%rsi
0xffffffff811f0116 <n_tty_read+751>: shr $0x3,%al
0xffffffff811f0119 <n_tty_read+754>: mov %eax,%ecx
0xffffffff811f011b <n_tty_read+756>: and $0x1,%ecx
0xffffffff811f011e <n_tty_read+759>: mov %ecx,-0x9c(%rbp)
0xffffffff811f0124 <n_tty_read+765>: callq 0xffffffff8106201b
<add_wait_queue>
0xffffffff811f0129 <n_tty_read+770>: movslq -0xb0(%rbp),%rbx
0xffffffff811f0130 <n_tty_read+777>: movslq -0xac(%rbp),%rax
0xffffffff811f0137 <n_tty_read+784>: mov -0xd8(%rbp),%rdx
0xffffffff811f013e <n_tty_read+791>: inc %rdx
0xffffffff811f0141 <n_tty_read+794>: mov %rbx,-0xe0(%rbp)
0xffffffff811f0148 <n_tty_read+801>: mov %rax,-0xe8(%rbp)
0xffffffff811f014f <n_tty_read+808>: mov %rdx,-0xf0(%rbp)
0xffffffff811f0156 <n_tty_read+815>: jmpq 0xffffffff811f0522
<n_tty_read+1787>
0xffffffff811f015b <n_tty_read+820>: cmpl $0x0,-0x9c(%rbp)
0xffffffff811f0162 <n_tty_read+827>: je 0xffffffff811f01ef
<n_tty_read+968>
0xffffffff811f0168 <n_tty_read+833>: mov 0xf8(%r13),%rax
0xffffffff811f016f <n_tty_read+840>: cmpb $0x0,0xed(%rax)
0xffffffff811f0176 <n_tty_read+847>: je 0xffffffff811f01ef
<n_tty_read+968>
0xffffffff811f0178 <n_tty_read+849>: mov -0xd8(%rbp),%rcx
0xffffffff811f017f <n_tty_read+856>: cmp %rcx,-0x40(%rbp)
0xffffffff811f0183 <n_tty_read+860>: jne 0xffffffff811f052d
<n_tty_read+1798>
0xffffffff811f0189 <n_tty_read+866>: lea 0x68(%rax),%rdi
0xffffffff811f018d <n_tty_read+870>: callq 0xffffffff812d6fb8
<_spin_lock_irqsave>
0xffffffff811f0192 <n_tty_read+875>: mov 0xf8(%r13),%rdi
0xffffffff811f0199 <n_tty_read+882>: mov %rax,%rsi
0xffffffff811f019c <n_tty_read+885>: mov 0xed(%rdi),%bl
0xffffffff811f01a2 <n_tty_read+891>: movb $0x0,0xed(%rdi)
0xffffffff811f01a9 <n_tty_read+898>: add $0x68,%rdi
0xffffffff811f01ad <n_tty_read+902>: callq 0xffffffff812d70c1
<_spin_unlock_irqrestore>
0xffffffff811f01b2 <n_tty_read+907>: mov -0x40(%rbp),%r12
0xffffffff811f01b6 <n_tty_read+911>: lea -0x31(%rbp),%rsi
0xffffffff811f01ba <n_tty_read+915>: mov $0x1,%edx
0xffffffff811f01bf <n_tty_read+920>: mov %r13,%rdi
0xffffffff811f01c2 <n_tty_read+923>: mov %bl,-0x31(%rbp)
0xffffffff811f01c5 <n_tty_read+926>: lea 0x1(%r12),%rax
0xffffffff811f01ca <n_tty_read+931>: mov %rax,-0x40(%rbp)
0xffffffff811f01ce <n_tty_read+935>: callq 0xffffffff812008ac
<tty_audit_add_data>
0xffffffff811f01d3 <n_tty_read+940>: mov -0x31(%rbp),%al
0xffffffff811f01d6 <n_tty_read+943>: mov %r12,%rcx
0xffffffff811f01d9 <n_tty_read+946>: callq 0xffffffff811949a0
<__put_user_1>
0xffffffff811f01de <n_tty_read+951>: test %eax,%eax
0xffffffff811f01e0 <n_tty_read+953>: jne 0xffffffff811f043d
<n_tty_read+1558>
0xffffffff811f01e6 <n_tty_read+959>: decq -0x78(%rbp)
0xffffffff811f01ea <n_tty_read+963>: jmpq 0xffffffff811f052d
<n_tty_read+1798>
0xffffffff811f01ef <n_tty_read+968>: mov $0x1,%eax
0xffffffff811f01f4 <n_tty_read+973>: mov -0x100(%rbp),%rbx
0xffffffff811f01fb <n_tty_read+980>: xchg %rax,(%rbx)
0xffffffff811f01fe <n_tty_read+983>: mov -0x40(%rbp),%rcx
0xffffffff811f0202 <n_tty_read+987>: mov -0xd8(%rbp),%rax
0xffffffff811f0209 <n_tty_read+994>: mov -0xe0(%rbp),%rbx
0xffffffff811f0210 <n_tty_read+1001>: sub %rcx,%rax
0xffffffff811f0213 <n_tty_read+1004>: lea (%rax,%rbx,1),%rdx
0xffffffff811f0217 <n_tty_read+1008>: movzwl 0x21e(%r13),%eax
0xffffffff811f021f <n_tty_read+1016>: cmp %rax,%rdx
0xffffffff811f0222 <n_tty_read+1019>: jge 0xffffffff811f0240
<n_tty_read+1049>
0xffffffff811f0224 <n_tty_read+1021>: test %rdx,%rdx
0xffffffff811f0227 <n_tty_read+1024>: jle 0xffffffff811f0240
<n_tty_read+1049>
0xffffffff811f0229 <n_tty_read+1026>: mov -0xd8(%rbp),%eax
0xffffffff811f022f <n_tty_read+1032>: sub %cx,%ax
0xffffffff811f0232 <n_tty_read+1035>: add -0xb0(%rbp),%eax
0xffffffff811f0238 <n_tty_read+1041>: mov %ax,0x21e(%r13)
0xffffffff811f0240 <n_tty_read+1049>: mov %r13,%rdi
0xffffffff811f0243 <n_tty_read+1052>: callq 0xffffffff811f37f3
<tty_flush_to_ldisc>
0xffffffff811f0248 <n_tty_read+1057>: testb $0x10,0x21c(%r13)
0xffffffff811f0250 <n_tty_read+1065>: je 0xffffffff811f0261
<n_tty_read+1082>
0xffffffff811f0252 <n_tty_read+1067>: cmpl $0x0,0x478(%r13)
0xffffffff811f025a <n_tty_read+1075>: jne 0xffffffff811f026f
<n_tty_read+1096>
0xffffffff811f025c <n_tty_read+1077>: jmpq 0xffffffff811f0621
<n_tty_read+2042>
0xffffffff811f0261 <n_tty_read+1082>: cmpl $0x0,0x260(%r13)
0xffffffff811f0269 <n_tty_read+1090>: jle 0xffffffff811f0621
<n_tty_read+2042>
0xffffffff811f026f <n_tty_read+1096>: mov -0x110(%rbp),%rax
0xffffffff811f0276 <n_tty_read+1103>: movq $0x0,(%rax)
0xffffffff811f027d <n_tty_read+1110>: cmpl $0x0,-0x9c(%rbp)
0xffffffff811f0284 <n_tty_read+1117>: mov -0x40(%rbp),%rax
0xffffffff811f0288 <n_tty_read+1121>: je 0xffffffff811f0376
<n_tty_read+1359>
0xffffffff811f028e <n_tty_read+1127>: cmp -0xd8(%rbp),%rax
0xffffffff811f0295 <n_tty_read+1134>: jne 0xffffffff811f0376
<n_tty_read+1359>
0xffffffff811f029b <n_tty_read+1140>: jmpq 0xffffffff811f033b
<n_tty_read+1300>
0xffffffff811f02a0 <n_tty_read+1145>: mov -0xd0(%rbp),%rdi
0xffffffff811f02a7 <n_tty_read+1152>: callq 0xffffffff811eb980
<tty_hung_up_p>
0xffffffff811f02ac <n_tty_read+1157>: test %eax,%eax
0xffffffff811f02ae <n_tty_read+1159>: jne 0xffffffff811f052d
<n_tty_read+1798>
0xffffffff811f02b4 <n_tty_read+1165>: test %r15,%r15
0xffffffff811f02b7 <n_tty_read+1168>: je 0xffffffff811f052d
<n_tty_read+1798>
0xffffffff811f02bd <n_tty_read+1174>: mov -0xd0(%rbp),%rdx
0xffffffff811f02c4 <n_tty_read+1181>: testb $0x8,0x39(%rdx)
0xffffffff811f02c8 <n_tty_read+1185>: je 0xffffffff811f02da
<n_tty_read+1203>
0xffffffff811f02ca <n_tty_read+1187>: movq
$0xfffffffffffffff5,-0xa8(%rbp)
0xffffffff811f02d5 <n_tty_read+1198>: jmpq 0xffffffff811f052d
<n_tty_read+1798>
0xffffffff811f02da <n_tty_read+1203>: mov -0x108(%rbp),%rcx
0xffffffff811f02e1 <n_tty_read+1210>: mov 0x8(%rcx),%rax
0xffffffff811f02e5 <n_tty_read+1214>: testb $0x4,0x10(%rax)
0xffffffff811f02e9 <n_tty_read+1218>: je 0xffffffff811f02fb
<n_tty_read+1236>
0xffffffff811f02eb <n_tty_read+1220>: movq
$0xfffffffffffffe00,-0xa8(%rbp)
0xffffffff811f02f6 <n_tty_read+1231>: jmpq 0xffffffff811f052d
<n_tty_read+1798>
0xffffffff811f02fb <n_tty_read+1236>: mov $0xfff,%eax
0xffffffff811f0300 <n_tty_read+1241>: sub 0x260(%r13),%eax
0xffffffff811f0307 <n_tty_read+1248>: test %eax,%eax
0xffffffff811f0309 <n_tty_read+1250>: jg 0xffffffff811f0324
<n_tty_read+1277>
0xffffffff811f030b <n_tty_read+1252>: xor %eax,%eax
0xffffffff811f030d <n_tty_read+1254>: testb $0x10,0x21c(%r13)
0xffffffff811f0315 <n_tty_read+1262>: je 0xffffffff811f0324
<n_tty_read+1277>
0xffffffff811f0317 <n_tty_read+1264>: xor %eax,%eax
0xffffffff811f0319 <n_tty_read+1266>: cmpl $0x0,0x478(%r13)
0xffffffff811f0321 <n_tty_read+1274>: sete %al
0xffffffff811f0324 <n_tty_read+1277>: mov %r15,%rdi
0xffffffff811f0327 <n_tty_read+1280>: mov %eax,0xf0(%r13)
0xffffffff811f032e <n_tty_read+1287>: callq 0xffffffff812d5a02
<schedule_timeout>
0xffffffff811f0333 <n_tty_read+1292>: mov %rax,%r15
0xffffffff811f0336 <n_tty_read+1295>: jmpq 0xffffffff811f0522
<n_tty_read+1787>
0xffffffff811f033b <n_tty_read+1300>: mov -0xf0(%rbp),%rbx
0xffffffff811f0342 <n_tty_read+1307>: lea -0x31(%rbp),%rsi
0xffffffff811f0346 <n_tty_read+1311>: mov $0x1,%edx
0xffffffff811f034b <n_tty_read+1316>: mov %r13,%rdi
0xffffffff811f034e <n_tty_read+1319>: movb $0x0,-0x31(%rbp)
0xffffffff811f0352 <n_tty_read+1323>: mov %rbx,-0x40(%rbp)
0xffffffff811f0356 <n_tty_read+1327>: callq 0xffffffff812008ac
<tty_audit_add_data>
0xffffffff811f035b <n_tty_read+1332>: mov -0x31(%rbp),%al
0xffffffff811f035e <n_tty_read+1335>: mov -0xd8(%rbp),%rcx
0xffffffff811f0365 <n_tty_read+1342>: callq 0xffffffff811949a0
<__put_user_1>
0xffffffff811f036a <n_tty_read+1347>: test %eax,%eax
0xffffffff811f036c <n_tty_read+1349>: jne 0xffffffff811f043d
<n_tty_read+1558>
0xffffffff811f0372 <n_tty_read+1355>: decq -0x78(%rbp)
0xffffffff811f0376 <n_tty_read+1359>: testb $0x10,0x21c(%r13)
0xffffffff811f037e <n_tty_read+1367>: jne 0xffffffff811f0456
<n_tty_read+1583>
0xffffffff811f0384 <n_tty_read+1373>: jmpq 0xffffffff811f047a
<n_tty_read+1619>
0xffffffff811f0389 <n_tty_read+1378>: mov 0x25c(%r13),%eax
0xffffffff811f0390 <n_tty_read+1385>: mov -0x88(%rbp),%rbx
0xffffffff811f0397 <n_tty_read+1392>: lock btr %eax,(%rbx)
0xffffffff811f039b <n_tty_read+1396>: sbb %r14d,%r14d
0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx
0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax
0xffffffff811f03ac <n_tty_read+1413>: mov -0xb8(%rbp),%rdi
0xffffffff811f03b3 <n_tty_read+1420>: movsbl (%rax,%rdx,1),%ebx
Below is the output of bt -a command in crash
bt -a
PID: 0 TASK: ffffffff814204b0 CPU: 0 COMMAND: "swapper"
#0 [ffff880033007e80] crash_nmi_callback at ffffffff8101fbc9
#1 [ffff880033007e90] notifier_call_chain at ffffffff81065893
#2 [ffff880033007ed0] atomic_notifier_call_chain at ffffffff810658dd
#3 [ffff880033007ee0] notify_die at ffffffff8106597f
#4 [ffff880033007f10] do_nmi at ffffffff8100dc5d
#5 [ffff880033007f50] nmi at ffffffff812d76b0
[exception RIP: mwait_idle+163]
RIP: ffffffff81013029 RSP: ffffffff813e3eb8 RFLAGS: 00000246
RAX: 0000000000000000 RBX: ffffffff813e3fd8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffffffff813e3fd8 RDI: ffffffff81522308
RBP: ffffffff813e3ec8 R8: 0000000000000000 R9: ffff88003306e290
R10: 0000000000012d80 R11: 0000000000000000 R12: ffffffff8147e368
R13: ffffffff814ccb30 R14: ffffffff814cdfa0 R15: ffffffff813e3fa8
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
--- <NMI exception stack> ---
#6 [ffffffff813e3eb8] mwait_idle at ffffffff81013029
#7 [ffffffff813e3ed0] cpu_idle at ffffffff8100af21
PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"
#0 [ffff88031ce759d0] machine_kexec at ffffffff81024486
#1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230
#2 [ffff88031ce75b20] oops_end at ffffffff8100fa38
#3 [ffff88031ce75b50] no_context at ffffffff8102d801
#4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9
#5 [ffff88031ce75c70] bad_area at ffffffff8102da41
#6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19
#7 [ffff88031ce75cf0] page_fault at ffffffff812d7425
[exception RIP: n_tty_read+1420]
RIP: ffffffff811f03b3 RSP: ffff88031ce75da8 RFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044
RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c
RBP: ffff88031ce75eb8 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044
R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#8 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e
#9 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8
#10 [ffff88031ce75f40] sys_read at ffffffff810ebe48
#11 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2
RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212
RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000
RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000
RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
PID: 0 TASK: ffff88031e0e3540 CPU: 2 COMMAND: "swapper"
#0 [ffff880033047e80] crash_nmi_callback at ffffffff8101fbc9
#1 [ffff880033047e90] notifier_call_chain at ffffffff81065893
#2 [ffff880033047ed0] atomic_notifier_call_chain at ffffffff810658dd
#3 [ffff880033047ee0] notify_die at ffffffff8106597f
#4 [ffff880033047f10] do_nmi at ffffffff8100dc5d
#5 [ffff880033047f50] nmi at ffffffff812d76b0
[exception RIP: mwait_idle+163]
RIP: ffffffff81013029 RSP: ffff88031e0e5ef8 RFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff88031e0e5fd8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88031e0e5fd8 RDI: ffffffff81522308
RBP: ffff88031e0e5f08 R8: 0000000000000000 R9: ffff88003302e290
R10: 0000000000012d80 R11: 0000000000000000 R12: ffffffff8147e368
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
--- <NMI exception stack> ---
#6 [ffff88031e0e5ef8] mwait_idle at ffffffff81013029
#7 [ffff88031e0e5f10] cpu_idle at ffffffff8100af21
PID: 0 TASK: ffff88031e113580 CPU: 3 COMMAND: "swapper"
#0 [ffff880033067e80] crash_nmi_callback at ffffffff8101fbc9
#1 [ffff880033067e90] notifier_call_chain at ffffffff81065893
#2 [ffff880033067ed0] atomic_notifier_call_chain at ffffffff810658dd
#3 [ffff880033067ee0] notify_die at ffffffff8106597f
#4 [ffff880033067f10] do_nmi at ffffffff8100dc5d
#5 [ffff880033067f50] nmi at ffffffff812d76b0
[exception RIP: mwait_idle+163]
RIP: ffffffff81013029 RSP: ffff88031e115ef8 RFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff88031e115fd8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88031e115fd8 RDI: ffffffff81522308
RBP: ffff88031e115f08 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000800 R11: 0000000000000000 R12: ffffffff8147e368
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
--- <NMI exception stack> ---
#6 [ffff88031e115ef8] mwait_idle at ffffffff81013029
#7 [ffff88031e115f10] cpu_idle at ffffffff8100af21
Please let me know if you need any other details.
Thanks and Regards
Shashidhara
-----Original Message-----
Sent: Thursday, June 23, 2011 9:35 PM
To: Discussion list for crash utility usage,maintenance and
development
Subject: Re: [Crash-utility] Unable to switch stack frames while using
crash
----- Original Message -----

Post by Dave Anderson
BTW, are you sure about that?
Presuming that the "tty" pointer is ffff8802cbd54800 as you've shown

below,

Post by Dave Anderson
and therefore tty->read_buf is 0xffff8802cbfe6000 and tty->read_tail

is 0,

Post by Dave Anderson
then the statement above would be simply be reading
tty->read_buf[0],

or

Post by Dave Anderson
virtual address 0xffff8802cbfe6000. But the oops shows it faulting
on

a

Post by Dave Anderson
BUG: unable to handle kernel NULL pointer dereference at

0000000000000005
Just for my own sanity, can you either attach the
"drivers/char/n_tty.c"
from *your* specific kernel, or get the source-code/line-number data
from
the embedded gdb module?
If you don't have the n_tty.c file readily available, you can get the
source-code/line-number data of a particular function by doing
something
Get the line number of the beginning of n_tty_read(), which in my
kernel
crash> gdb list n_tty_read
1695 * This code must be sure never to sleep through a hangup.
1696 */
1697
1698 static ssize_t n_tty_read(struct tty_struct *tty, struct file
*file,
1699 unsigned char __user *buf, size_t nr)
1700 {
1701 unsigned char __user *b = buf;
1702 DECLARE_WAITQUEUE(wait, current);
1703 int c;
1704 int minimum, time;
crash>
Then get the line number of the next function in the file, which is
crash> gdb list n_tty_write
1918 * lock themselves)
1919 */
1920
1921 static ssize_t n_tty_write(struct tty_struct *tty, struct file
*file,
1922 const unsigned char *buf, size_t nr)
1923 {
1924 const unsigned char *b = buf;
1925 DECLARE_WAITQUEUE(wait, current);
1926 int c;
1927 ssize_t retval = 0;
crash> gdb list 1698,1920
1698 static ssize_t n_tty_read(struct tty_struct *tty, struct file
*file,
1699 unsigned char __user *buf, size_t nr)
1700 {
1701 unsigned char __user *b = buf;
1702 DECLARE_WAITQUEUE(wait, current);
1703 int c;
1704 int minimum, time;
1705 ssize_t retval = 0;
1706 ssize_t size;
1707 long timeout;
1708 unsigned long flags;
1709 int packet;
1710
1712
1713 BUG_ON(!tty->read_buf);
1714
1715 c = job_control(tty, file);
1716 if (c < 0)
1717 return c;
1718
1719 minimum = time = 0;
1720 timeout = MAX_SCHEDULE_TIMEOUT;
1721 if (!tty->icanon) {
1722 time = (HZ / 10) * TIME_CHAR(tty);
1723 minimum = MIN_CHAR(tty);
...
And lastly, since the crash occurred at
IP: [<ffffffff811f03b3>] n_tty_read+0x58c/0x818
crash> dis -rl n_tty_read+0x58c
...
And then post all of that data.
Dave
--
Crash-utility mailing list
https://www.redhat.com/mailman/listinfo/crash-utility
Information transmitted by this e-mail is proprietary to MphasiS, its
associated companies and/ or its customers and is intended
for use only by the individual or entity to which it is addressed, and
may contain information that is privileged, confidential or
exempt from disclosure under applicable law. If you are not the
intended recipient or it appears that this mail has been forwarded
to you without proper authority, you are notified that any use or
dissemination of this information in any manner is strictly
prohibited. In such cases, please notify us immediately at
--
Crash-utility mailing list
https://www.redhat.com/mailman/listinfo/crash-utility

[Andrew Suffield]

Post by Dave Anderson
And for that matter, since tty->read_cnt is 0 above,
your original question as to how that code path was taken to
begin with is also valid.

Surely that implies tty->read_cnt has been modified since it was tested, and
hence you're looking at concurrency issues?

[Shashidhara Shamaiah]

Hi Dave,

Thanks again for the response.

When I check the address of tty->read_buf using kmem command in crash,
below is the output. Which shows that the memory is still not freed. If
the tty_close handler had been invoked, before accessing tty->read_buf
we should not get this output. Please correct me if I am erring.

kmem 0xffff8802cbfe6000
CACHE NAME OBJSIZE ALLOCATED TOTAL
SLABS SSIZE
ffff88031f8039c0 size-4096 4096 6322 6393
6393 4k
SLAB MEMORY TOTAL ALLOCATED FREE
ffff8802ac4811c0 ffff8802cbfe6000 1 1 0
FREE / [ALLOCATED]
[ffff8802cbfe6000]

PAGE PHYSICAL MAPPING INDEX CNT FLAGS
ffffea0009c9fa50 2cbfe6000 0 0 1 200000000000080


I would like to thank every one for your time and effort in analyzing
this issue.

Thanks and Regards
Shashidhara


-----Original Message-----
From: crash-utility-***@redhat.com
[mailto:crash-utility-***@redhat.com] On Behalf Of Dave Anderson
Sent: Friday, June 24, 2011 7:10 PM
To: Discussion list for crash utility usage,maintenance and development
Subject: Re: [Crash-utility] Unable to switch stack frames while using
crash


---- Original Message -----

Post by Dave Anderson

Post by Dave Anderson
BTW, are you sure about that?
Presuming that the "tty" pointer is ffff8802cbd54800 as you've shown below,
and therefore tty->read_buf is 0xffff8802cbfe6000 and tty->read_tail is 0,
then the statement above would be simply be reading tty->read_buf[0], or
virtual address 0xffff8802cbfe6000. But the oops shows it faulting on a

Well, as it turns out, you have every reason to be sure about that...

Anyway, I don't understand why line numbers are not available with

Post by Dave Anderson
crash> dis -rl n_tty_read+0x58c
dis: line numbers are not available

... [ cut ] ...

Post by Dave Anderson
0xffffffff811f039b <n_tty_read+1396>: sbb %r14d,%r14d
0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx
0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax
0xffffffff811f03ac <n_tty_read+1413>: mov -0xb8(%rbp),%rdi
0xffffffff811f03b3 <n_tty_read+1420>: movsbl (%rax,%rdx,1),%ebx

But nonetheless, there is only on movsbl instruction in n_tty_read(),
and looking at a RHEL6 kernel, you were correct in your original
determination of the faulting instruction:

crash> dis n_tty_read | grep movsbl
0xffffffff812f88c9 <n_tty_read+0x2c9>: movsbl (%rdx,%rax,1),%ebx
crash> dis -rl 0xffffffff812f88c9 | tail
... [ cut ] ...
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/drivers/char/n_tty.c:
1821
0xffffffff812f88c9 <n_tty_read+0x2c9>: movsbl (%rdx,%rax,1),%ebx
crash>

which is here:

1814 if (tty->icanon) {
1815 /* N.B. avoid overrun if nr == 0 */
1816 while (nr && tty->read_cnt) {
1817 int eol;
1818
1819 eol =
test_and_clear_bit(tty->read_tail,
1820
tty->read_flags);
1821 c =
tty->read_buf[tty->read_tail];

The tty_struct offsets are these:

crash> tty_struct -o
struct tty_struct {
... [ cut ]...
[0x250] char *read_buf;
[0x258] int read_head;
[0x25c] int read_tail;
...

And you can see in the previous instructions the tty->read_buf (0x250)
and tty->read_tail (0x25c) offsets being added to the tty_struct

Post by Dave Anderson
0xffffffff811f039b <n_tty_read+1396>: sbb %r14d,%r14d
0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx
0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax
0xffffffff811f03ac <n_tty_read+1413>: mov -0xb8(%rbp),%rdi
0xffffffff811f03b3 <n_tty_read+1420>: movsbl (%rax,%rdx,1),%ebx

And as you originally reported, the tty_struct address in R13
is ffff8802cbd54800:

PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"
#0 [ffff88031ce759d0] machine_kexec at ffffffff81024486
#1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230
#2 [ffff88031ce75b20] oops_end at ffffffff8100fa38
#3 [ffff88031ce75b50] no_context at ffffffff8102d801
#4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9
#5 [ffff88031ce75c70] bad_area at ffffffff8102da41
#6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19
#7 [ffff88031ce75cf0] page_fault at ffffffff812d7425
[exception RIP: n_tty_read+1420]
RIP: ffffffff811f03b3 RSP: ffff88031ce75da8 RFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044
RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c
RBP: ffff88031ce75eb8 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044
R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#8 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e
#9 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8
#10 [ffff88031ce75f40] sys_read at ffffffff810ebe48
#11 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2
RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212
RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000
RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000
RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b

But for whatever reason -- and I cannot explain it -- after these

Post by Dave Anderson
0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx
0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax

the RDX register ended up with 0000000000000005, and the RAX register
with
a 0000000000000000, leading to the:

BUG: unable to handle kernel NULL pointer dereference at
0000000000000005

But when you display the tty_struct at ffff8802cbd54800, you see the
read_buf and read_tail with seemingly legitimate values:

crash> tty_struct ffff8802cbd54800
struct tty_struct { ...
magic = 21505,
driver = 0xffff88031b54ea00,
ops = 0xffffffff8130f650,
name = "pts9\000\...",
driver_data = 0xffff88029c8a9668,
icanon = 1 '\001',
read_buf = 0xffff8802cbfe6000 "",
read_head = 0,
read_tail = 0,
read_cnt = 0,
...

So everything in your analysis was correct, but how it is possible
that the RDX and RAX registers to have ended up with 0 and 5 is hard
to explain. And for that matter, since tty->read_cnt is 0 above,
your original question as to how that code path was taken to
begin with is also valid.

So I don't know -- anybody on the list ever seen anything like this?

Stumped,
Dave


----- Original Message -----

Post by Dave Anderson
Hi Dave,
Thank you so much for your help.
Below is the output of dis -rl n_tty_read+0x58c
crash> dis -rl n_tty_read+0x58c
dis: line numbers are not available
0xffffffff811efe27 <n_tty_read>: push %rbp
0xffffffff811efe28 <n_tty_read+1>: mov %gs:0xb500,%rax
0xffffffff811efe31 <n_tty_read+10>: mov %rsp,%rbp
0xffffffff811efe34 <n_tty_read+13>: push %r15
0xffffffff811efe36 <n_tty_read+15>: push %r14
0xffffffff811efe38 <n_tty_read+17>: push %r13
0xffffffff811efe3a <n_tty_read+19>: mov %rdi,%r13
0xffffffff811efe3d <n_tty_read+22>: lea -0x70(%rbp),%rdi
0xffffffff811efe41 <n_tty_read+26>: push %r12
0xffffffff811efe43 <n_tty_read+28>: push %rbx
0xffffffff811efe44 <n_tty_read+29>: lea 0x490(%r13),%rbx
0xffffffff811efe4b <n_tty_read+36>: sub $0xe8,%rsp
0xffffffff811efe52 <n_tty_read+43>: mov %rax,-0x98(%rbp)
0xffffffff811efe59 <n_tty_read+50>: mov %rcx,-0x78(%rbp)
0xffffffff811efe5d <n_tty_read+54>: xor %eax,%eax
0xffffffff811efe5f <n_tty_read+56>: mov $0xa,%ecx
0xffffffff811efe64 <n_tty_read+61>: mov %rdx,-0xd8(%rbp)
0xffffffff811efe6b <n_tty_read+68>: mov %rsi,-0xd0(%rbp)
0xffffffff811efe72 <n_tty_read+75>: mov %rdx,-0x40(%rbp)
0xffffffff811efe76 <n_tty_read+79>: rep stos %eax,%es:(%rdi)
0xffffffff811efe78 <n_tty_read+81>: lea 0x1c0(%r13),%rax
0xffffffff811efe7f <n_tty_read+88>: lea 0x1c8(%r13),%rcx
0xffffffff811efe86 <n_tty_read+95>: mov %rbx,-0xc0(%rbp)
0xffffffff811efe8d <n_tty_read+102>: lea 0xd8(%r13),%rbx
0xffffffff811efe94 <n_tty_read+109>: movq
$0xffffffff81045f84,-0x60(%rbp)
0xffffffff811efe9c <n_tty_read+117>: movq $0x0,-0xa8(%rbp)
0xffffffff811efea7 <n_tty_read+128>: mov -0x98(%rbp),%rdx
0xffffffff811efeae <n_tty_read+135>: mov %rax,-0xc8(%rbp)
0xffffffff811efeb5 <n_tty_read+142>: mov -0x98(%rbp),%rax
0xffffffff811efebc <n_tty_read+149>: mov %rcx,-0x90(%rbp)
0xffffffff811efec3 <n_tty_read+156>: lea 0x51c(%r13),%rcx
0xffffffff811efeca <n_tty_read+163>: mov %rbx,-0x80(%rbp)
0xffffffff811efece <n_tty_read+167>: mov %rdx,-0x68(%rbp)
0xffffffff811efed2 <n_tty_read+171>: lea 0x268(%r13),%rdx
0xffffffff811efed9 <n_tty_read+178>: mov %rcx,-0xb8(%rbp)
0xffffffff811efee0 <n_tty_read+185>: mov %rax,-0xf8(%rbp)
0xffffffff811efee7 <n_tty_read+192>: mov %rax,-0x100(%rbp)
0xffffffff811efeee <n_tty_read+199>: mov %rdx,-0x88(%rbp)
0xffffffff811efef5 <n_tty_read+206>: mov %rax,-0x108(%rbp)
0xffffffff811efefc <n_tty_read+213>: mov %rax,-0x110(%rbp)
0xffffffff811eff03 <n_tty_read+220>: cmpq $0x0,0x250(%r13)
0xffffffff811eff0b <n_tty_read+228>: jne 0xffffffff811eff11
<n_tty_read+234>
0xffffffff811eff0d <n_tty_read+230>: ud2a
0xffffffff811eff0f <n_tty_read+232>: jmp 0xffffffff811eff0f
<n_tty_read+232>
0xffffffff811eff11 <n_tty_read+234>: mov -0xd0(%rbp),%rdx
0xffffffff811eff18 <n_tty_read+241>: mov 0x20(%rdx),%rax
0xffffffff811eff1c <n_tty_read+245>: cmpq
$0xffffffff811ed61f,0x18(%rax)
0xffffffff811eff24 <n_tty_read+253>: je 0xffffffff811effef
<n_tty_read+456>
0xffffffff811eff2a <n_tty_read+259>: mov -0xf8(%rbp),%rcx
0xffffffff811eff31 <n_tty_read+266>: mov 0x478(%rcx),%rax
0xffffffff811eff38 <n_tty_read+273>: cmp %r13,0x180(%rax)
0xffffffff811eff3f <n_tty_read+280>: jne 0xffffffff811effef
<n_tty_read+456>
0xffffffff811eff45 <n_tty_read+286>: mov 0xc8(%r13),%rdx
0xffffffff811eff4c <n_tty_read+293>: test %rdx,%rdx
0xffffffff811eff4f <n_tty_read+296>: jne 0xffffffff811eff64
<n_tty_read+317>
0xffffffff811eff51 <n_tty_read+298>: mov $0xffffffff8139c972,%rdi
0xffffffff811eff58 <n_tty_read+305>: xor %eax,%eax
0xffffffff811eff5a <n_tty_read+307>: callq 0xffffffff812d4abf
<printk>
0xffffffff811eff5f <n_tty_read+312>: jmpq 0xffffffff811effef
<n_tty_read+456>
0xffffffff811eff64 <n_tty_read+317>: mov -0xf8(%rbp),%rbx
0xffffffff811eff6b <n_tty_read+324>: mov 0x1e0(%rbx),%rax
0xffffffff811eff72 <n_tty_read+331>: cmp %rdx,0x238(%rax)
0xffffffff811eff79 <n_tty_read+338>: je 0xffffffff811effef
<n_tty_read+456>
0xffffffff811eff7b <n_tty_read+340>: mov -0x98(%rbp),%rax
0xffffffff811eff82 <n_tty_read+347>: testb $0x10,0x48a(%rax)
0xffffffff811eff89 <n_tty_read+354>: jne 0xffffffff811f0611
<n_tty_read+2026>
0xffffffff811eff8f <n_tty_read+360>: mov 0x480(%rax),%rax
0xffffffff811eff96 <n_tty_read+367>: cmpq $0x1,0x288(%rax)
0xffffffff811eff9e <n_tty_read+375>: jne 0xffffffff811f0604
<n_tty_read+2013>
0xffffffff811effa4 <n_tty_read+381>: jmpq 0xffffffff811f0611
<n_tty_read+2026>
0xffffffff811effa9 <n_tty_read+386>: mov -0x98(%rbp),%rcx
0xffffffff811effb0 <n_tty_read+393>: mov $0x1,%edx
0xffffffff811effb5 <n_tty_read+398>: mov $0x15,%esi
0xffffffff811effba <n_tty_read+403>: mov 0x1e0(%rcx),%rax
0xffffffff811effc1 <n_tty_read+410>: mov 0x238(%rax),%rdi
0xffffffff811effc8 <n_tty_read+417>: callq 0xffffffff8105953a
<kill_pgrp>
0xffffffff811effcd <n_tty_read+422>: mov %gs:0xb508,%rdx
0xffffffff811effd6 <n_tty_read+431>: lea -0x1fc8(%rdx),%rax
0xffffffff811effdd <n_tty_read+438>: lock orb $0x4,-0x1fc8(%rdx)
0xffffffff811effe5 <n_tty_read+446>: mov $0xfffffe00,%eax
0xffffffff811effea <n_tty_read+451>: jmpq 0xffffffff811f0616
<n_tty_read+2031>
0xffffffff811effef <n_tty_read+456>: testb $0x10,0x21c(%r13)
0xffffffff811efff7 <n_tty_read+464>: je 0xffffffff811f000f
<n_tty_read+488>
0xffffffff811efff9 <n_tty_read+466>: movl $0x0,-0xb0(%rbp)
0xffffffff811f0003 <n_tty_read+476>: movl $0x0,-0xac(%rbp)
0xffffffff811f000d <n_tty_read+486>: jmp 0xffffffff811f0076
<n_tty_read+591>
0xffffffff811f000f <n_tty_read+488>: mov 0x70(%r13),%rdx
0xffffffff811f0013 <n_tty_read+492>: movzbl 0x16(%rdx),%eax
0xffffffff811f0017 <n_tty_read+496>: imul $0x19,%eax,%eax
0xffffffff811f001a <n_tty_read+499>: mov %eax,-0xac(%rbp)
0xffffffff811f0020 <n_tty_read+505>: movzbl 0x17(%rdx),%edx
0xffffffff811f0024 <n_tty_read+509>: test %edx,%edx
0xffffffff811f0026 <n_tty_read+511>: mov %edx,-0xb0(%rbp)
0xffffffff811f002c <n_tty_read+517>: je 0xffffffff811f0082
<n_tty_read+603>
0xffffffff811f002e <n_tty_read+519>: test %eax,%eax
0xffffffff811f0030 <n_tty_read+521>: je 0xffffffff811f003e
<n_tty_read+535>
0xffffffff811f0032 <n_tty_read+523>: movw $0x1,0x21e(%r13)
0xffffffff811f003c <n_tty_read+533>: jmp 0xffffffff811f0076
<n_tty_read+591>
0xffffffff811f003e <n_tty_read+535>: mov -0x90(%rbp),%rbx
0xffffffff811f0045 <n_tty_read+542>: cmp %rbx,0x1c8(%r13)
0xffffffff811f004c <n_tty_read+549>: je 0xffffffff811f0068
<n_tty_read+577>
0xffffffff811f004e <n_tty_read+551>: movzwl 0x21e(%r13),%eax
0xffffffff811f0056 <n_tty_read+559>: mov $0x7fffffffffffffff,%r15
0xffffffff811f0060 <n_tty_read+569>: cmp -0xb0(%rbp),%eax
0xffffffff811f0066 <n_tty_read+575>: jle 0xffffffff811f00b7
<n_tty_read+656>
0xffffffff811f0068 <n_tty_read+577>: mov -0xb0(%rbp),%eax
0xffffffff811f006e <n_tty_read+583>: mov %ax,0x21e(%r13)
0xffffffff811f0076 <n_tty_read+591>: mov $0x7fffffffffffffff,%r15
0xffffffff811f0080 <n_tty_read+601>: jmp 0xffffffff811f00b7
<n_tty_read+656>
0xffffffff811f0082 <n_tty_read+603>: movslq -0xac(%rbp),%r15
0xffffffff811f0089 <n_tty_read+610>: cmpl $0x0,-0xac(%rbp)
0xffffffff811f0090 <n_tty_read+617>: mov $0x0,%eax
0xffffffff811f0095 <n_tty_read+622>: movw $0x1,0x21e(%r13)
0xffffffff811f009f <n_tty_read+632>: movl $0x1,-0xb0(%rbp)
0xffffffff811f00a9 <n_tty_read+642>: movl $0x0,-0xac(%rbp)
0xffffffff811f00b3 <n_tty_read+652>: cmove %rax,%r15
0xffffffff811f00b7 <n_tty_read+656>: mov -0xd0(%rbp),%rdx
0xffffffff811f00be <n_tty_read+663>: testb $0x8,0x39(%rdx)
0xffffffff811f00c2 <n_tty_read+667>: je 0xffffffff811f00e4
<n_tty_read+701>
0xffffffff811f00c4 <n_tty_read+669>: mov -0xc0(%rbp),%rdi
0xffffffff811f00cb <n_tty_read+676>: callq 0xffffffff812d5ec7
<mutex_trylock>
0xffffffff811f00d0 <n_tty_read+681>: test %eax,%eax
0xffffffff811f00d2 <n_tty_read+683>: jne 0xffffffff811f0104
<n_tty_read+733>
0xffffffff811f00d4 <n_tty_read+685>: movq
$0xfffffffffffffff5,-0xa8(%rbp)
0xffffffff811f00df <n_tty_read+696>: jmpq 0xffffffff811f05eb
<n_tty_read+1988>
0xffffffff811f00e4 <n_tty_read+701>: mov -0xc0(%rbp),%rdi
0xffffffff811f00eb <n_tty_read+708>: callq 0xffffffff812d6358
<mutex_lock_interruptible>
0xffffffff811f00f0 <n_tty_read+713>: test %eax,%eax
0xffffffff811f00f2 <n_tty_read+715>: je 0xffffffff811f0104
<n_tty_read+733>
0xffffffff811f00f4 <n_tty_read+717>: movq
$0xfffffffffffffe00,-0xa8(%rbp)
0xffffffff811f00ff <n_tty_read+728>: jmpq 0xffffffff811f05eb
<n_tty_read+1988>
0xffffffff811f0104 <n_tty_read+733>: mov 0xec(%r13),%al
0xffffffff811f010b <n_tty_read+740>: mov -0xc8(%rbp),%rdi
0xffffffff811f0112 <n_tty_read+747>: lea -0x70(%rbp),%rsi
0xffffffff811f0116 <n_tty_read+751>: shr $0x3,%al
0xffffffff811f0119 <n_tty_read+754>: mov %eax,%ecx
0xffffffff811f011b <n_tty_read+756>: and $0x1,%ecx
0xffffffff811f011e <n_tty_read+759>: mov %ecx,-0x9c(%rbp)
0xffffffff811f0124 <n_tty_read+765>: callq 0xffffffff8106201b
<add_wait_queue>
0xffffffff811f0129 <n_tty_read+770>: movslq -0xb0(%rbp),%rbx
0xffffffff811f0130 <n_tty_read+777>: movslq -0xac(%rbp),%rax
0xffffffff811f0137 <n_tty_read+784>: mov -0xd8(%rbp),%rdx
0xffffffff811f013e <n_tty_read+791>: inc %rdx
0xffffffff811f0141 <n_tty_read+794>: mov %rbx,-0xe0(%rbp)
0xffffffff811f0148 <n_tty_read+801>: mov %rax,-0xe8(%rbp)
0xffffffff811f014f <n_tty_read+808>: mov %rdx,-0xf0(%rbp)
0xffffffff811f0156 <n_tty_read+815>: jmpq 0xffffffff811f0522
<n_tty_read+1787>
0xffffffff811f015b <n_tty_read+820>: cmpl $0x0,-0x9c(%rbp)
0xffffffff811f0162 <n_tty_read+827>: je 0xffffffff811f01ef
<n_tty_read+968>
0xffffffff811f0168 <n_tty_read+833>: mov 0xf8(%r13),%rax
0xffffffff811f016f <n_tty_read+840>: cmpb $0x0,0xed(%rax)
0xffffffff811f0176 <n_tty_read+847>: je 0xffffffff811f01ef
<n_tty_read+968>
0xffffffff811f0178 <n_tty_read+849>: mov -0xd8(%rbp),%rcx
0xffffffff811f017f <n_tty_read+856>: cmp %rcx,-0x40(%rbp)
0xffffffff811f0183 <n_tty_read+860>: jne 0xffffffff811f052d
<n_tty_read+1798>
0xffffffff811f0189 <n_tty_read+866>: lea 0x68(%rax),%rdi
0xffffffff811f018d <n_tty_read+870>: callq 0xffffffff812d6fb8
<_spin_lock_irqsave>
0xffffffff811f0192 <n_tty_read+875>: mov 0xf8(%r13),%rdi
0xffffffff811f0199 <n_tty_read+882>: mov %rax,%rsi
0xffffffff811f019c <n_tty_read+885>: mov 0xed(%rdi),%bl
0xffffffff811f01a2 <n_tty_read+891>: movb $0x0,0xed(%rdi)
0xffffffff811f01a9 <n_tty_read+898>: add $0x68,%rdi
0xffffffff811f01ad <n_tty_read+902>: callq 0xffffffff812d70c1
<_spin_unlock_irqrestore>
0xffffffff811f01b2 <n_tty_read+907>: mov -0x40(%rbp),%r12
0xffffffff811f01b6 <n_tty_read+911>: lea -0x31(%rbp),%rsi
0xffffffff811f01ba <n_tty_read+915>: mov $0x1,%edx
0xffffffff811f01bf <n_tty_read+920>: mov %r13,%rdi
0xffffffff811f01c2 <n_tty_read+923>: mov %bl,-0x31(%rbp)
0xffffffff811f01c5 <n_tty_read+926>: lea 0x1(%r12),%rax
0xffffffff811f01ca <n_tty_read+931>: mov %rax,-0x40(%rbp)
0xffffffff811f01ce <n_tty_read+935>: callq 0xffffffff812008ac
<tty_audit_add_data>
0xffffffff811f01d3 <n_tty_read+940>: mov -0x31(%rbp),%al
0xffffffff811f01d6 <n_tty_read+943>: mov %r12,%rcx
0xffffffff811f01d9 <n_tty_read+946>: callq 0xffffffff811949a0
<__put_user_1>
0xffffffff811f01de <n_tty_read+951>: test %eax,%eax
0xffffffff811f01e0 <n_tty_read+953>: jne 0xffffffff811f043d
<n_tty_read+1558>
0xffffffff811f01e6 <n_tty_read+959>: decq -0x78(%rbp)
0xffffffff811f01ea <n_tty_read+963>: jmpq 0xffffffff811f052d
<n_tty_read+1798>
0xffffffff811f01ef <n_tty_read+968>: mov $0x1,%eax
0xffffffff811f01f4 <n_tty_read+973>: mov -0x100(%rbp),%rbx
0xffffffff811f01fb <n_tty_read+980>: xchg %rax,(%rbx)
0xffffffff811f01fe <n_tty_read+983>: mov -0x40(%rbp),%rcx
0xffffffff811f0202 <n_tty_read+987>: mov -0xd8(%rbp),%rax
0xffffffff811f0209 <n_tty_read+994>: mov -0xe0(%rbp),%rbx
0xffffffff811f0210 <n_tty_read+1001>: sub %rcx,%rax
0xffffffff811f0213 <n_tty_read+1004>: lea (%rax,%rbx,1),%rdx
0xffffffff811f0217 <n_tty_read+1008>: movzwl 0x21e(%r13),%eax
0xffffffff811f021f <n_tty_read+1016>: cmp %rax,%rdx
0xffffffff811f0222 <n_tty_read+1019>: jge 0xffffffff811f0240
<n_tty_read+1049>
0xffffffff811f0224 <n_tty_read+1021>: test %rdx,%rdx
0xffffffff811f0227 <n_tty_read+1024>: jle 0xffffffff811f0240
<n_tty_read+1049>
0xffffffff811f0229 <n_tty_read+1026>: mov -0xd8(%rbp),%eax
0xffffffff811f022f <n_tty_read+1032>: sub %cx,%ax
0xffffffff811f0232 <n_tty_read+1035>: add -0xb0(%rbp),%eax
0xffffffff811f0238 <n_tty_read+1041>: mov %ax,0x21e(%r13)
0xffffffff811f0240 <n_tty_read+1049>: mov %r13,%rdi
0xffffffff811f0243 <n_tty_read+1052>: callq 0xffffffff811f37f3
<tty_flush_to_ldisc>
0xffffffff811f0248 <n_tty_read+1057>: testb $0x10,0x21c(%r13)
0xffffffff811f0250 <n_tty_read+1065>: je 0xffffffff811f0261
<n_tty_read+1082>
0xffffffff811f0252 <n_tty_read+1067>: cmpl $0x0,0x478(%r13)
0xffffffff811f025a <n_tty_read+1075>: jne 0xffffffff811f026f
<n_tty_read+1096>
0xffffffff811f025c <n_tty_read+1077>: jmpq 0xffffffff811f0621
<n_tty_read+2042>
0xffffffff811f0261 <n_tty_read+1082>: cmpl $0x0,0x260(%r13)
0xffffffff811f0269 <n_tty_read+1090>: jle 0xffffffff811f0621
<n_tty_read+2042>
0xffffffff811f026f <n_tty_read+1096>: mov -0x110(%rbp),%rax
0xffffffff811f0276 <n_tty_read+1103>: movq $0x0,(%rax)
0xffffffff811f027d <n_tty_read+1110>: cmpl $0x0,-0x9c(%rbp)
0xffffffff811f0284 <n_tty_read+1117>: mov -0x40(%rbp),%rax
0xffffffff811f0288 <n_tty_read+1121>: je 0xffffffff811f0376
<n_tty_read+1359>
0xffffffff811f028e <n_tty_read+1127>: cmp -0xd8(%rbp),%rax
0xffffffff811f0295 <n_tty_read+1134>: jne 0xffffffff811f0376
<n_tty_read+1359>
0xffffffff811f029b <n_tty_read+1140>: jmpq 0xffffffff811f033b
<n_tty_read+1300>
0xffffffff811f02a0 <n_tty_read+1145>: mov -0xd0(%rbp),%rdi
0xffffffff811f02a7 <n_tty_read+1152>: callq 0xffffffff811eb980
<tty_hung_up_p>
0xffffffff811f02ac <n_tty_read+1157>: test %eax,%eax
0xffffffff811f02ae <n_tty_read+1159>: jne 0xffffffff811f052d
<n_tty_read+1798>
0xffffffff811f02b4 <n_tty_read+1165>: test %r15,%r15
0xffffffff811f02b7 <n_tty_read+1168>: je 0xffffffff811f052d
<n_tty_read+1798>
0xffffffff811f02bd <n_tty_read+1174>: mov -0xd0(%rbp),%rdx
0xffffffff811f02c4 <n_tty_read+1181>: testb $0x8,0x39(%rdx)
0xffffffff811f02c8 <n_tty_read+1185>: je 0xffffffff811f02da
<n_tty_read+1203>
0xffffffff811f02ca <n_tty_read+1187>: movq
$0xfffffffffffffff5,-0xa8(%rbp)
0xffffffff811f02d5 <n_tty_read+1198>: jmpq 0xffffffff811f052d
<n_tty_read+1798>
0xffffffff811f02da <n_tty_read+1203>: mov -0x108(%rbp),%rcx
0xffffffff811f02e1 <n_tty_read+1210>: mov 0x8(%rcx),%rax
0xffffffff811f02e5 <n_tty_read+1214>: testb $0x4,0x10(%rax)
0xffffffff811f02e9 <n_tty_read+1218>: je 0xffffffff811f02fb
<n_tty_read+1236>
0xffffffff811f02eb <n_tty_read+1220>: movq
$0xfffffffffffffe00,-0xa8(%rbp)
0xffffffff811f02f6 <n_tty_read+1231>: jmpq 0xffffffff811f052d
<n_tty_read+1798>
0xffffffff811f02fb <n_tty_read+1236>: mov $0xfff,%eax
0xffffffff811f0300 <n_tty_read+1241>: sub 0x260(%r13),%eax
0xffffffff811f0307 <n_tty_read+1248>: test %eax,%eax
0xffffffff811f0309 <n_tty_read+1250>: jg 0xffffffff811f0324
<n_tty_read+1277>
0xffffffff811f030b <n_tty_read+1252>: xor %eax,%eax
0xffffffff811f030d <n_tty_read+1254>: testb $0x10,0x21c(%r13)
0xffffffff811f0315 <n_tty_read+1262>: je 0xffffffff811f0324
<n_tty_read+1277>
0xffffffff811f0317 <n_tty_read+1264>: xor %eax,%eax
0xffffffff811f0319 <n_tty_read+1266>: cmpl $0x0,0x478(%r13)
0xffffffff811f0321 <n_tty_read+1274>: sete %al
0xffffffff811f0324 <n_tty_read+1277>: mov %r15,%rdi
0xffffffff811f0327 <n_tty_read+1280>: mov %eax,0xf0(%r13)
0xffffffff811f032e <n_tty_read+1287>: callq 0xffffffff812d5a02
<schedule_timeout>
0xffffffff811f0333 <n_tty_read+1292>: mov %rax,%r15
0xffffffff811f0336 <n_tty_read+1295>: jmpq 0xffffffff811f0522
<n_tty_read+1787>
0xffffffff811f033b <n_tty_read+1300>: mov -0xf0(%rbp),%rbx
0xffffffff811f0342 <n_tty_read+1307>: lea -0x31(%rbp),%rsi
0xffffffff811f0346 <n_tty_read+1311>: mov $0x1,%edx
0xffffffff811f034b <n_tty_read+1316>: mov %r13,%rdi
0xffffffff811f034e <n_tty_read+1319>: movb $0x0,-0x31(%rbp)
0xffffffff811f0352 <n_tty_read+1323>: mov %rbx,-0x40(%rbp)
0xffffffff811f0356 <n_tty_read+1327>: callq 0xffffffff812008ac
<tty_audit_add_data>
0xffffffff811f035b <n_tty_read+1332>: mov -0x31(%rbp),%al
0xffffffff811f035e <n_tty_read+1335>: mov -0xd8(%rbp),%rcx
0xffffffff811f0365 <n_tty_read+1342>: callq 0xffffffff811949a0
<__put_user_1>
0xffffffff811f036a <n_tty_read+1347>: test %eax,%eax
0xffffffff811f036c <n_tty_read+1349>: jne 0xffffffff811f043d
<n_tty_read+1558>
0xffffffff811f0372 <n_tty_read+1355>: decq -0x78(%rbp)
0xffffffff811f0376 <n_tty_read+1359>: testb $0x10,0x21c(%r13)
0xffffffff811f037e <n_tty_read+1367>: jne 0xffffffff811f0456
<n_tty_read+1583>
0xffffffff811f0384 <n_tty_read+1373>: jmpq 0xffffffff811f047a
<n_tty_read+1619>
0xffffffff811f0389 <n_tty_read+1378>: mov 0x25c(%r13),%eax
0xffffffff811f0390 <n_tty_read+1385>: mov -0x88(%rbp),%rbx
0xffffffff811f0397 <n_tty_read+1392>: lock btr %eax,(%rbx)
0xffffffff811f039b <n_tty_read+1396>: sbb %r14d,%r14d
0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx
0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax
0xffffffff811f03ac <n_tty_read+1413>: mov -0xb8(%rbp),%rdi
0xffffffff811f03b3 <n_tty_read+1420>: movsbl (%rax,%rdx,1),%ebx
Below is the output of bt -a command in crash
bt -a
PID: 0 TASK: ffffffff814204b0 CPU: 0 COMMAND: "swapper"
#0 [ffff880033007e80] crash_nmi_callback at ffffffff8101fbc9
#1 [ffff880033007e90] notifier_call_chain at ffffffff81065893
#2 [ffff880033007ed0] atomic_notifier_call_chain at ffffffff810658dd
#3 [ffff880033007ee0] notify_die at ffffffff8106597f
#4 [ffff880033007f10] do_nmi at ffffffff8100dc5d
#5 [ffff880033007f50] nmi at ffffffff812d76b0
[exception RIP: mwait_idle+163]
RIP: ffffffff81013029 RSP: ffffffff813e3eb8 RFLAGS: 00000246
RAX: 0000000000000000 RBX: ffffffff813e3fd8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffffffff813e3fd8 RDI: ffffffff81522308
RBP: ffffffff813e3ec8 R8: 0000000000000000 R9: ffff88003306e290
R10: 0000000000012d80 R11: 0000000000000000 R12: ffffffff8147e368
R13: ffffffff814ccb30 R14: ffffffff814cdfa0 R15: ffffffff813e3fa8
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
--- <NMI exception stack> ---
#6 [ffffffff813e3eb8] mwait_idle at ffffffff81013029
#7 [ffffffff813e3ed0] cpu_idle at ffffffff8100af21
PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"
#0 [ffff88031ce759d0] machine_kexec at ffffffff81024486
#1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230
#2 [ffff88031ce75b20] oops_end at ffffffff8100fa38
#3 [ffff88031ce75b50] no_context at ffffffff8102d801
#4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9
#5 [ffff88031ce75c70] bad_area at ffffffff8102da41
#6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19
#7 [ffff88031ce75cf0] page_fault at ffffffff812d7425
[exception RIP: n_tty_read+1420]
RIP: ffffffff811f03b3 RSP: ffff88031ce75da8 RFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044
RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c
RBP: ffff88031ce75eb8 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044
R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#8 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e
#9 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8
#10 [ffff88031ce75f40] sys_read at ffffffff810ebe48
#11 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2
RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212
RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000
RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000
RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
PID: 0 TASK: ffff88031e0e3540 CPU: 2 COMMAND: "swapper"
#0 [ffff880033047e80] crash_nmi_callback at ffffffff8101fbc9
#1 [ffff880033047e90] notifier_call_chain at ffffffff81065893
#2 [ffff880033047ed0] atomic_notifier_call_chain at ffffffff810658dd
#3 [ffff880033047ee0] notify_die at ffffffff8106597f
#4 [ffff880033047f10] do_nmi at ffffffff8100dc5d
#5 [ffff880033047f50] nmi at ffffffff812d76b0
[exception RIP: mwait_idle+163]
RIP: ffffffff81013029 RSP: ffff88031e0e5ef8 RFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff88031e0e5fd8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88031e0e5fd8 RDI: ffffffff81522308
RBP: ffff88031e0e5f08 R8: 0000000000000000 R9: ffff88003302e290
R10: 0000000000012d80 R11: 0000000000000000 R12: ffffffff8147e368
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
--- <NMI exception stack> ---
#6 [ffff88031e0e5ef8] mwait_idle at ffffffff81013029
#7 [ffff88031e0e5f10] cpu_idle at ffffffff8100af21
PID: 0 TASK: ffff88031e113580 CPU: 3 COMMAND: "swapper"
#0 [ffff880033067e80] crash_nmi_callback at ffffffff8101fbc9
#1 [ffff880033067e90] notifier_call_chain at ffffffff81065893
#2 [ffff880033067ed0] atomic_notifier_call_chain at ffffffff810658dd
#3 [ffff880033067ee0] notify_die at ffffffff8106597f
#4 [ffff880033067f10] do_nmi at ffffffff8100dc5d
#5 [ffff880033067f50] nmi at ffffffff812d76b0
[exception RIP: mwait_idle+163]
RIP: ffffffff81013029 RSP: ffff88031e115ef8 RFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff88031e115fd8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88031e115fd8 RDI: ffffffff81522308
RBP: ffff88031e115f08 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000800 R11: 0000000000000000 R12: ffffffff8147e368
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
--- <NMI exception stack> ---
#6 [ffff88031e115ef8] mwait_idle at ffffffff81013029
#7 [ffff88031e115f10] cpu_idle at ffffffff8100af21
Please let me know if you need any other details.
Thanks and Regards
Shashidhara
-----Original Message-----
Sent: Thursday, June 23, 2011 9:35 PM
To: Discussion list for crash utility usage,maintenance and
development
Subject: Re: [Crash-utility] Unable to switch stack frames while using
crash
----- Original Message -----

Post by Dave Anderson
BTW, are you sure about that?
Presuming that the "tty" pointer is ffff8802cbd54800 as you've shown

below,

Post by Dave Anderson
and therefore tty->read_buf is 0xffff8802cbfe6000 and tty->read_tail

is 0,

Post by Dave Anderson
then the statement above would be simply be reading
tty->read_buf[0],

or

Post by Dave Anderson
virtual address 0xffff8802cbfe6000. But the oops shows it faulting
on

a

Post by Dave Anderson
BUG: unable to handle kernel NULL pointer dereference at

0000000000000005
Just for my own sanity, can you either attach the
"drivers/char/n_tty.c"
from *your* specific kernel, or get the source-code/line-number data
from
the embedded gdb module?
If you don't have the n_tty.c file readily available, you can get the
source-code/line-number data of a particular function by doing
something
Get the line number of the beginning of n_tty_read(), which in my kernel
crash> gdb list n_tty_read
1695 * This code must be sure never to sleep through a hangup.
1696 */
1697
1698 static ssize_t n_tty_read(struct tty_struct *tty, struct file
*file,
1699 unsigned char __user *buf, size_t nr)
1700 {
1701 unsigned char __user *b = buf;
1702 DECLARE_WAITQUEUE(wait, current);
1703 int c;
1704 int minimum, time;
crash>
Then get the line number of the next function in the file, which is
crash> gdb list n_tty_write
1918 * lock themselves)
1919 */
1920
1921 static ssize_t n_tty_write(struct tty_struct *tty, struct file
*file,
1922 const unsigned char *buf, size_t nr)
1923 {
1924 const unsigned char *b = buf;
1925 DECLARE_WAITQUEUE(wait, current);
1926 int c;
1927 ssize_t retval = 0;
crash> gdb list 1698,1920
1698 static ssize_t n_tty_read(struct tty_struct *tty, struct file
*file,
1699 unsigned char __user *buf, size_t nr)
1700 {
1701 unsigned char __user *b = buf;
1702 DECLARE_WAITQUEUE(wait, current);
1703 int c;
1704 int minimum, time;
1705 ssize_t retval = 0;
1706 ssize_t size;
1707 long timeout;
1708 unsigned long flags;
1709 int packet;
1710
1712
1713 BUG_ON(!tty->read_buf);
1714
1715 c = job_control(tty, file);
1716 if (c < 0)
1717 return c;
1718
1719 minimum = time = 0;
1720 timeout = MAX_SCHEDULE_TIMEOUT;
1721 if (!tty->icanon) {
1722 time = (HZ / 10) * TIME_CHAR(tty);
1723 minimum = MIN_CHAR(tty);
...
And lastly, since the crash occurred at
IP: [<ffffffff811f03b3>] n_tty_read+0x58c/0x818
crash> dis -rl n_tty_read+0x58c
...
And then post all of that data.
Dave
--
Crash-utility mailing list
https://www.redhat.com/mailman/listinfo/crash-utility
Information transmitted by this e-mail is proprietary to MphasiS, its
associated companies and/ or its customers and is intended
for use only by the individual or entity to which it is addressed, and
may contain information that is privileged, confidential or
exempt from disclosure under applicable law. If you are not the
intended recipient or it appears that this mail has been forwarded
to you without proper authority, you are notified that any use or
dissemination of this information in any manner is strictly
prohibited. In such cases, please notify us immediately at
--
Crash-utility mailing list
https://www.redhat.com/mailman/listinfo/crash-utility

--
Crash-utility mailing list
Crash-***@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Information transmitted by this e-mail is proprietary to MphasiS, its associated companies and/ or its customers and is intended
for use only by the individual or entity to which it is addressed, and may contain information that is privileged, confidential or
exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded
to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly
prohibited. In such cases, please notify us immediately at ***@mphasis.com and delete this mail from your records.

[Dave Anderson]

----- Original Message -----

Post by Dave Anderson
And for that matter, since tty->read_cnt is 0 above,
your original question as to how that code path was taken to
begin with is also valid.
Surely that implies tty->read_cnt has been modified since it was
tested, and hence you're looking at concurrency issues?

Yeah, although the contents tty->read_buf are hard to explain.
It gets allocated during n_tty_open() and freed during n_tty_close().
And at the beginning of n_tty_read() there's:

BUG_ON(!tty->read_buf);

and the dump-time contents show a buffer allocated:

crash> tty_struct ffff8802cbd54800
struct tty_struct { ...
magic = 21505,
driver = 0xffff88031b54ea00,
ops = 0xffffffff8130f650,
name = "pts9\000\...",
driver_data = 0xffff88029c8a9668,
icanon = 1 '\001',
read_buf = 0xffff8802cbfe6000 "",
read_head = 0,
read_tail = 0,
read_cnt = 0,
...

but it's a NULL pointer when read during the function?

Dave

[Andrew Suffield]

Post by Dave Anderson
Yeah, although the contents tty->read_buf are hard to explain.
It gets allocated during n_tty_open() and freed during n_tty_close().
BUG_ON(!tty->read_buf);
crash> tty_struct ffff8802cbd54800
struct tty_struct { ...
magic = 21505,
driver = 0xffff88031b54ea00,
ops = 0xffffffff8130f650,
name = "pts9\000\...",
driver_data = 0xffff88029c8a9668,
icanon = 1 '\001',
read_buf = 0xffff8802cbfe6000 "",
read_head = 0,
read_tail = 0,
read_cnt = 0,
...
but it's a NULL pointer when read during the function?

Hmm, that is interesting. Assuming that we are in fact dealing with a
software bug where this memory area changed recently, the only possible
explanation I can see is that n_tty_close() has been called while
n_tty_read() is in progress. I know that's not supposed to happen, but the
implication would be that is the bug - a locking issue in the tty layer that
allows them to be closed while calls are in progress. Sadly that code isn't
as mature as it should be and there was a whole load of concurrency issues
fixed in the late 2.6.30s; I don't know the details, but it might be one of
those.

Alternatively it's regular old hardware failure and we're just looking at
junk.

I think that's about as far as we can go on the information available (and
also the extent to which it's relevant to this mailing list)

[Dave Anderson]

----- Original Message -----

Post by Shashidhara Shamaiah
Hi Dave,
Thanks again for the response.
When I check the address of tty->read_buf using kmem command in crash,
below is the output. Which shows that the memory is still not freed. If
the tty_close handler had been invoked, before accessing tty->read_buf
we should not get this output. Please correct me if I am erring.
kmem 0xffff8802cbfe6000
CACHE NAME OBJSIZE ALLOCATED TOTAL SLABS SSIZE
ffff88031f8039c0 size-4096 4096 6322 6393 6393 4k
SLAB MEMORY TOTAL ALLOCATED FREE
ffff8802ac4811c0 ffff8802cbfe6000 1 1 0
FREE / [ALLOCATED]
[ffff8802cbfe6000]
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
ffffea0009c9fa50 2cbfe6000 0 0 1 200000000000080
I would like to thank every one for your time and effort in analyzing
this issue.
Thanks and Regards
Shashidhara

That particular 4k buffer was definitely allocated at the point
in time when the crash occurred.

However, it doesn't make logical sense because the code path saw
a non-zero read_cnt and a NULL read_buf pointer. So if by some
chance there were an illegal intervening n_tty_close() call, it
would have had to have happened:

(1) after tty->read_cnt was read, and
(2) before the "c = " assignment (because read_buf was NULL), and
(3) then would have to have been re-opened, reallocated the buffer
that's being seen in the dump.

while (nr && tty->read_cnt) {
int eol;

eol = test_and_clear_bit(tty->read_tail,
tty->read_flags);
c = tty->read_buf[tty->read_tail];
spin_lock_irqsave(&tty->read_lock, flags);

So it just seems impossible to recreate even with an intervening
n_tty_close().

Dave



Dave

Post by Shashidhara Shamaiah
-----Original Message-----
Sent: Friday, June 24, 2011 7:10 PM
To: Discussion list for crash utility usage,maintenance and
development
Subject: Re: [Crash-utility] Unable to switch stack frames while using
crash
---- Original Message -----

Post by Dave Anderson

Post by Dave Anderson
BTW, are you sure about that?
Presuming that the "tty" pointer is ffff8802cbd54800 as you've
shown

below,

Post by Dave Anderson

Post by Dave Anderson
and therefore tty->read_buf is 0xffff8802cbfe6000 and
tty->read_tail

is 0,

Post by Dave Anderson

Post by Dave Anderson
then the statement above would be simply be reading
tty->read_buf[0],

or

Post by Dave Anderson

Post by Dave Anderson
virtual address 0xffff8802cbfe6000. But the oops shows it faulting

on a
Well, as it turns out, you have every reason to be sure about that...
Anyway, I don't understand why line numbers are not available with

Post by Dave Anderson
crash> dis -rl n_tty_read+0x58c
dis: line numbers are not available

... [ cut ] ...

Post by Dave Anderson
0xffffffff811f039b <n_tty_read+1396>: sbb %r14d,%r14d
0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx
0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax
0xffffffff811f03ac <n_tty_read+1413>: mov -0xb8(%rbp),%rdi
0xffffffff811f03b3 <n_tty_read+1420>: movsbl (%rax,%rdx,1),%ebx

But nonetheless, there is only on movsbl instruction in n_tty_read(),
and looking at a RHEL6 kernel, you were correct in your original
crash> dis n_tty_read | grep movsbl
0xffffffff812f88c9 <n_tty_read+0x2c9>: movsbl (%rdx,%rax,1),%ebx
crash> dis -rl 0xffffffff812f88c9 | tail
... [ cut ] ...
1821
0xffffffff812f88c9 <n_tty_read+0x2c9>: movsbl (%rdx,%rax,1),%ebx
crash>
1814 if (tty->icanon) {
1815 /* N.B. avoid overrun if nr == 0 */
1816 while (nr && tty->read_cnt) {
1817 int eol;
1818
1819 eol =
test_and_clear_bit(tty->read_tail,
1820
tty->read_flags);
1821 c =
tty->read_buf[tty->read_tail];
crash> tty_struct -o
struct tty_struct {
... [ cut ]...
[0x250] char *read_buf;
[0x258] int read_head;
[0x25c] int read_tail;
...
And you can see in the previous instructions the tty->read_buf (0x250)
and tty->read_tail (0x25c) offsets being added to the tty_struct

Post by Dave Anderson
0xffffffff811f039b <n_tty_read+1396>: sbb %r14d,%r14d
0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx
0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax
0xffffffff811f03ac <n_tty_read+1413>: mov -0xb8(%rbp),%rdi
0xffffffff811f03b3 <n_tty_read+1420>: movsbl (%rax,%rdx,1),%ebx

And as you originally reported, the tty_struct address in R13
PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"
#0 [ffff88031ce759d0] machine_kexec at ffffffff81024486
#1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230
#2 [ffff88031ce75b20] oops_end at ffffffff8100fa38
#3 [ffff88031ce75b50] no_context at ffffffff8102d801
#4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9
#5 [ffff88031ce75c70] bad_area at ffffffff8102da41
#6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19
#7 [ffff88031ce75cf0] page_fault at ffffffff812d7425
[exception RIP: n_tty_read+1420]
RIP: ffffffff811f03b3 RSP: ffff88031ce75da8 RFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044
RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c
RBP: ffff88031ce75eb8 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044
R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#8 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e
#9 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8
#10 [ffff88031ce75f40] sys_read at ffffffff810ebe48
#11 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2
RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212
RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000
RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000
RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
But for whatever reason -- and I cannot explain it -- after these

Post by Dave Anderson
0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx
0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax

the RDX register ended up with 0000000000000005, and the RAX register
with
BUG: unable to handle kernel NULL pointer dereference at
0000000000000005
But when you display the tty_struct at ffff8802cbd54800, you see the
crash> tty_struct ffff8802cbd54800
struct tty_struct { ...
magic = 21505,
driver = 0xffff88031b54ea00,
ops = 0xffffffff8130f650,
name = "pts9\000\...",
driver_data = 0xffff88029c8a9668,
icanon = 1 '\001',
read_buf = 0xffff8802cbfe6000 "",
read_head = 0,
read_tail = 0,
read_cnt = 0,
...
So everything in your analysis was correct, but how it is possible
that the RDX and RAX registers to have ended up with 0 and 5 is hard
to explain. And for that matter, since tty->read_cnt is 0 above,
your original question as to how that code path was taken to
begin with is also valid.
So I don't know -- anybody on the list ever seen anything like this?
Stumped,
Dave
----- Original Message -----

Post by Dave Anderson
Hi Dave,
Thank you so much for your help.
Below is the output of dis -rl n_tty_read+0x58c
crash> dis -rl n_tty_read+0x58c
dis: line numbers are not available
0xffffffff811efe27 <n_tty_read>: push %rbp
0xffffffff811efe28 <n_tty_read+1>: mov %gs:0xb500,%rax
0xffffffff811efe31 <n_tty_read+10>: mov %rsp,%rbp
0xffffffff811efe34 <n_tty_read+13>: push %r15
0xffffffff811efe36 <n_tty_read+15>: push %r14
0xffffffff811efe38 <n_tty_read+17>: push %r13
0xffffffff811efe3a <n_tty_read+19>: mov %rdi,%r13
0xffffffff811efe3d <n_tty_read+22>: lea -0x70(%rbp),%rdi
0xffffffff811efe41 <n_tty_read+26>: push %r12
0xffffffff811efe43 <n_tty_read+28>: push %rbx
0xffffffff811efe44 <n_tty_read+29>: lea 0x490(%r13),%rbx
0xffffffff811efe4b <n_tty_read+36>: sub $0xe8,%rsp
0xffffffff811efe52 <n_tty_read+43>: mov %rax,-0x98(%rbp)
0xffffffff811efe59 <n_tty_read+50>: mov %rcx,-0x78(%rbp)
0xffffffff811efe5d <n_tty_read+54>: xor %eax,%eax
0xffffffff811efe5f <n_tty_read+56>: mov $0xa,%ecx
0xffffffff811efe64 <n_tty_read+61>: mov %rdx,-0xd8(%rbp)
0xffffffff811efe6b <n_tty_read+68>: mov %rsi,-0xd0(%rbp)
0xffffffff811efe72 <n_tty_read+75>: mov %rdx,-0x40(%rbp)
0xffffffff811efe76 <n_tty_read+79>: rep stos %eax,%es:(%rdi)
0xffffffff811efe78 <n_tty_read+81>: lea 0x1c0(%r13),%rax
0xffffffff811efe7f <n_tty_read+88>: lea 0x1c8(%r13),%rcx
0xffffffff811efe86 <n_tty_read+95>: mov %rbx,-0xc0(%rbp)
0xffffffff811efe8d <n_tty_read+102>: lea 0xd8(%r13),%rbx
0xffffffff811efe94 <n_tty_read+109>: movq
$0xffffffff81045f84,-0x60(%rbp)
0xffffffff811efe9c <n_tty_read+117>: movq $0x0,-0xa8(%rbp)
0xffffffff811efea7 <n_tty_read+128>: mov -0x98(%rbp),%rdx
0xffffffff811efeae <n_tty_read+135>: mov %rax,-0xc8(%rbp)
0xffffffff811efeb5 <n_tty_read+142>: mov -0x98(%rbp),%rax
0xffffffff811efebc <n_tty_read+149>: mov %rcx,-0x90(%rbp)
0xffffffff811efec3 <n_tty_read+156>: lea 0x51c(%r13),%rcx
0xffffffff811efeca <n_tty_read+163>: mov %rbx,-0x80(%rbp)
0xffffffff811efece <n_tty_read+167>: mov %rdx,-0x68(%rbp)
0xffffffff811efed2 <n_tty_read+171>: lea 0x268(%r13),%rdx
0xffffffff811efed9 <n_tty_read+178>: mov %rcx,-0xb8(%rbp)
0xffffffff811efee0 <n_tty_read+185>: mov %rax,-0xf8(%rbp)
0xffffffff811efee7 <n_tty_read+192>: mov %rax,-0x100(%rbp)
0xffffffff811efeee <n_tty_read+199>: mov %rdx,-0x88(%rbp)
0xffffffff811efef5 <n_tty_read+206>: mov %rax,-0x108(%rbp)
0xffffffff811efefc <n_tty_read+213>: mov %rax,-0x110(%rbp)
0xffffffff811eff03 <n_tty_read+220>: cmpq $0x0,0x250(%r13)
0xffffffff811eff0b <n_tty_read+228>: jne 0xffffffff811eff11
<n_tty_read+234>
0xffffffff811eff0d <n_tty_read+230>: ud2a
0xffffffff811eff0f <n_tty_read+232>: jmp 0xffffffff811eff0f
<n_tty_read+232>
0xffffffff811eff11 <n_tty_read+234>: mov -0xd0(%rbp),%rdx
0xffffffff811eff18 <n_tty_read+241>: mov 0x20(%rdx),%rax
0xffffffff811eff1c <n_tty_read+245>: cmpq
$0xffffffff811ed61f,0x18(%rax)
0xffffffff811eff24 <n_tty_read+253>: je 0xffffffff811effef
<n_tty_read+456>
0xffffffff811eff2a <n_tty_read+259>: mov -0xf8(%rbp),%rcx
0xffffffff811eff31 <n_tty_read+266>: mov 0x478(%rcx),%rax
0xffffffff811eff38 <n_tty_read+273>: cmp %r13,0x180(%rax)
0xffffffff811eff3f <n_tty_read+280>: jne 0xffffffff811effef
<n_tty_read+456>
0xffffffff811eff45 <n_tty_read+286>: mov 0xc8(%r13),%rdx
0xffffffff811eff4c <n_tty_read+293>: test %rdx,%rdx
0xffffffff811eff4f <n_tty_read+296>: jne 0xffffffff811eff64
<n_tty_read+317>
0xffffffff811eff51 <n_tty_read+298>: mov $0xffffffff8139c972,%rdi
0xffffffff811eff58 <n_tty_read+305>: xor %eax,%eax
0xffffffff811eff5a <n_tty_read+307>: callq 0xffffffff812d4abf
<printk>
0xffffffff811eff5f <n_tty_read+312>: jmpq 0xffffffff811effef
<n_tty_read+456>
0xffffffff811eff64 <n_tty_read+317>: mov -0xf8(%rbp),%rbx
0xffffffff811eff6b <n_tty_read+324>: mov 0x1e0(%rbx),%rax
0xffffffff811eff72 <n_tty_read+331>: cmp %rdx,0x238(%rax)
0xffffffff811eff79 <n_tty_read+338>: je 0xffffffff811effef
<n_tty_read+456>
0xffffffff811eff7b <n_tty_read+340>: mov -0x98(%rbp),%rax
0xffffffff811eff82 <n_tty_read+347>: testb $0x10,0x48a(%rax)
0xffffffff811eff89 <n_tty_read+354>: jne 0xffffffff811f0611
<n_tty_read+2026>
0xffffffff811eff8f <n_tty_read+360>: mov 0x480(%rax),%rax
0xffffffff811eff96 <n_tty_read+367>: cmpq $0x1,0x288(%rax)
0xffffffff811eff9e <n_tty_read+375>: jne 0xffffffff811f0604
<n_tty_read+2013>
0xffffffff811effa4 <n_tty_read+381>: jmpq 0xffffffff811f0611
<n_tty_read+2026>
0xffffffff811effa9 <n_tty_read+386>: mov -0x98(%rbp),%rcx
0xffffffff811effb0 <n_tty_read+393>: mov $0x1,%edx
0xffffffff811effb5 <n_tty_read+398>: mov $0x15,%esi
0xffffffff811effba <n_tty_read+403>: mov 0x1e0(%rcx),%rax
0xffffffff811effc1 <n_tty_read+410>: mov 0x238(%rax),%rdi
0xffffffff811effc8 <n_tty_read+417>: callq 0xffffffff8105953a
<kill_pgrp>
0xffffffff811effcd <n_tty_read+422>: mov %gs:0xb508,%rdx
0xffffffff811effd6 <n_tty_read+431>: lea -0x1fc8(%rdx),%rax
0xffffffff811effdd <n_tty_read+438>: lock orb $0x4,-0x1fc8(%rdx)
0xffffffff811effe5 <n_tty_read+446>: mov $0xfffffe00,%eax
0xffffffff811effea <n_tty_read+451>: jmpq 0xffffffff811f0616
<n_tty_read+2031>
0xffffffff811effef <n_tty_read+456>: testb $0x10,0x21c(%r13)
0xffffffff811efff7 <n_tty_read+464>: je 0xffffffff811f000f
<n_tty_read+488>
0xffffffff811efff9 <n_tty_read+466>: movl $0x0,-0xb0(%rbp)
0xffffffff811f0003 <n_tty_read+476>: movl $0x0,-0xac(%rbp)
0xffffffff811f000d <n_tty_read+486>: jmp 0xffffffff811f0076
<n_tty_read+591>
0xffffffff811f000f <n_tty_read+488>: mov 0x70(%r13),%rdx
0xffffffff811f0013 <n_tty_read+492>: movzbl 0x16(%rdx),%eax
0xffffffff811f0017 <n_tty_read+496>: imul $0x19,%eax,%eax
0xffffffff811f001a <n_tty_read+499>: mov %eax,-0xac(%rbp)
0xffffffff811f0020 <n_tty_read+505>: movzbl 0x17(%rdx),%edx
0xffffffff811f0024 <n_tty_read+509>: test %edx,%edx
0xffffffff811f0026 <n_tty_read+511>: mov %edx,-0xb0(%rbp)
0xffffffff811f002c <n_tty_read+517>: je 0xffffffff811f0082
<n_tty_read+603>
0xffffffff811f002e <n_tty_read+519>: test %eax,%eax
0xffffffff811f0030 <n_tty_read+521>: je 0xffffffff811f003e
<n_tty_read+535>
0xffffffff811f0032 <n_tty_read+523>: movw $0x1,0x21e(%r13)
0xffffffff811f003c <n_tty_read+533>: jmp 0xffffffff811f0076
<n_tty_read+591>
0xffffffff811f003e <n_tty_read+535>: mov -0x90(%rbp),%rbx
0xffffffff811f0045 <n_tty_read+542>: cmp %rbx,0x1c8(%r13)
0xffffffff811f004c <n_tty_read+549>: je 0xffffffff811f0068
<n_tty_read+577>
0xffffffff811f004e <n_tty_read+551>: movzwl 0x21e(%r13),%eax
0xffffffff811f0056 <n_tty_read+559>: mov $0x7fffffffffffffff,%r15
0xffffffff811f0060 <n_tty_read+569>: cmp -0xb0(%rbp),%eax
0xffffffff811f0066 <n_tty_read+575>: jle 0xffffffff811f00b7
<n_tty_read+656>
0xffffffff811f0068 <n_tty_read+577>: mov -0xb0(%rbp),%eax
0xffffffff811f006e <n_tty_read+583>: mov %ax,0x21e(%r13)
0xffffffff811f0076 <n_tty_read+591>: mov $0x7fffffffffffffff,%r15
0xffffffff811f0080 <n_tty_read+601>: jmp 0xffffffff811f00b7
<n_tty_read+656>
0xffffffff811f0082 <n_tty_read+603>: movslq -0xac(%rbp),%r15
0xffffffff811f0089 <n_tty_read+610>: cmpl $0x0,-0xac(%rbp)
0xffffffff811f0090 <n_tty_read+617>: mov $0x0,%eax
0xffffffff811f0095 <n_tty_read+622>: movw $0x1,0x21e(%r13)
0xffffffff811f009f <n_tty_read+632>: movl $0x1,-0xb0(%rbp)
0xffffffff811f00a9 <n_tty_read+642>: movl $0x0,-0xac(%rbp)
0xffffffff811f00b3 <n_tty_read+652>: cmove %rax,%r15
0xffffffff811f00b7 <n_tty_read+656>: mov -0xd0(%rbp),%rdx
0xffffffff811f00be <n_tty_read+663>: testb $0x8,0x39(%rdx)
0xffffffff811f00c2 <n_tty_read+667>: je 0xffffffff811f00e4
<n_tty_read+701>
0xffffffff811f00c4 <n_tty_read+669>: mov -0xc0(%rbp),%rdi
0xffffffff811f00cb <n_tty_read+676>: callq 0xffffffff812d5ec7
<mutex_trylock>
0xffffffff811f00d0 <n_tty_read+681>: test %eax,%eax
0xffffffff811f00d2 <n_tty_read+683>: jne 0xffffffff811f0104
<n_tty_read+733>
0xffffffff811f00d4 <n_tty_read+685>: movq
$0xfffffffffffffff5,-0xa8(%rbp)
0xffffffff811f00df <n_tty_read+696>: jmpq 0xffffffff811f05eb
<n_tty_read+1988>
0xffffffff811f00e4 <n_tty_read+701>: mov -0xc0(%rbp),%rdi
0xffffffff811f00eb <n_tty_read+708>: callq 0xffffffff812d6358
<mutex_lock_interruptible>
0xffffffff811f00f0 <n_tty_read+713>: test %eax,%eax
0xffffffff811f00f2 <n_tty_read+715>: je 0xffffffff811f0104
<n_tty_read+733>
0xffffffff811f00f4 <n_tty_read+717>: movq
$0xfffffffffffffe00,-0xa8(%rbp)
0xffffffff811f00ff <n_tty_read+728>: jmpq 0xffffffff811f05eb
<n_tty_read+1988>
0xffffffff811f0104 <n_tty_read+733>: mov 0xec(%r13),%al
0xffffffff811f010b <n_tty_read+740>: mov -0xc8(%rbp),%rdi
0xffffffff811f0112 <n_tty_read+747>: lea -0x70(%rbp),%rsi
0xffffffff811f0116 <n_tty_read+751>: shr $0x3,%al
0xffffffff811f0119 <n_tty_read+754>: mov %eax,%ecx
0xffffffff811f011b <n_tty_read+756>: and $0x1,%ecx
0xffffffff811f011e <n_tty_read+759>: mov %ecx,-0x9c(%rbp)
0xffffffff811f0124 <n_tty_read+765>: callq 0xffffffff8106201b
<add_wait_queue>
0xffffffff811f0129 <n_tty_read+770>: movslq -0xb0(%rbp),%rbx
0xffffffff811f0130 <n_tty_read+777>: movslq -0xac(%rbp),%rax
0xffffffff811f0137 <n_tty_read+784>: mov -0xd8(%rbp),%rdx
0xffffffff811f013e <n_tty_read+791>: inc %rdx
0xffffffff811f0141 <n_tty_read+794>: mov %rbx,-0xe0(%rbp)
0xffffffff811f0148 <n_tty_read+801>: mov %rax,-0xe8(%rbp)
0xffffffff811f014f <n_tty_read+808>: mov %rdx,-0xf0(%rbp)
0xffffffff811f0156 <n_tty_read+815>: jmpq 0xffffffff811f0522
<n_tty_read+1787>
0xffffffff811f015b <n_tty_read+820>: cmpl $0x0,-0x9c(%rbp)
0xffffffff811f0162 <n_tty_read+827>: je 0xffffffff811f01ef
<n_tty_read+968>
0xffffffff811f0168 <n_tty_read+833>: mov 0xf8(%r13),%rax
0xffffffff811f016f <n_tty_read+840>: cmpb $0x0,0xed(%rax)
0xffffffff811f0176 <n_tty_read+847>: je 0xffffffff811f01ef
<n_tty_read+968>
0xffffffff811f0178 <n_tty_read+849>: mov -0xd8(%rbp),%rcx
0xffffffff811f017f <n_tty_read+856>: cmp %rcx,-0x40(%rbp)
0xffffffff811f0183 <n_tty_read+860>: jne 0xffffffff811f052d
<n_tty_read+1798>
0xffffffff811f0189 <n_tty_read+866>: lea 0x68(%rax),%rdi
0xffffffff811f018d <n_tty_read+870>: callq 0xffffffff812d6fb8
<_spin_lock_irqsave>
0xffffffff811f0192 <n_tty_read+875>: mov 0xf8(%r13),%rdi
0xffffffff811f0199 <n_tty_read+882>: mov %rax,%rsi
0xffffffff811f019c <n_tty_read+885>: mov 0xed(%rdi),%bl
0xffffffff811f01a2 <n_tty_read+891>: movb $0x0,0xed(%rdi)
0xffffffff811f01a9 <n_tty_read+898>: add $0x68,%rdi
0xffffffff811f01ad <n_tty_read+902>: callq 0xffffffff812d70c1
<_spin_unlock_irqrestore>
0xffffffff811f01b2 <n_tty_read+907>: mov -0x40(%rbp),%r12
0xffffffff811f01b6 <n_tty_read+911>: lea -0x31(%rbp),%rsi
0xffffffff811f01ba <n_tty_read+915>: mov $0x1,%edx
0xffffffff811f01bf <n_tty_read+920>: mov %r13,%rdi
0xffffffff811f01c2 <n_tty_read+923>: mov %bl,-0x31(%rbp)
0xffffffff811f01c5 <n_tty_read+926>: lea 0x1(%r12),%rax
0xffffffff811f01ca <n_tty_read+931>: mov %rax,-0x40(%rbp)
0xffffffff811f01ce <n_tty_read+935>: callq 0xffffffff812008ac
<tty_audit_add_data>
0xffffffff811f01d3 <n_tty_read+940>: mov -0x31(%rbp),%al
0xffffffff811f01d6 <n_tty_read+943>: mov %r12,%rcx
0xffffffff811f01d9 <n_tty_read+946>: callq 0xffffffff811949a0
<__put_user_1>
0xffffffff811f01de <n_tty_read+951>: test %eax,%eax
0xffffffff811f01e0 <n_tty_read+953>: jne 0xffffffff811f043d
<n_tty_read+1558>
0xffffffff811f01e6 <n_tty_read+959>: decq -0x78(%rbp)
0xffffffff811f01ea <n_tty_read+963>: jmpq 0xffffffff811f052d
<n_tty_read+1798>
0xffffffff811f01ef <n_tty_read+968>: mov $0x1,%eax
0xffffffff811f01f4 <n_tty_read+973>: mov -0x100(%rbp),%rbx
0xffffffff811f01fb <n_tty_read+980>: xchg %rax,(%rbx)
0xffffffff811f01fe <n_tty_read+983>: mov -0x40(%rbp),%rcx
0xffffffff811f0202 <n_tty_read+987>: mov -0xd8(%rbp),%rax
0xffffffff811f0209 <n_tty_read+994>: mov -0xe0(%rbp),%rbx
0xffffffff811f0210 <n_tty_read+1001>: sub %rcx,%rax
0xffffffff811f0213 <n_tty_read+1004>: lea (%rax,%rbx,1),%rdx
0xffffffff811f0217 <n_tty_read+1008>: movzwl 0x21e(%r13),%eax
0xffffffff811f021f <n_tty_read+1016>: cmp %rax,%rdx
0xffffffff811f0222 <n_tty_read+1019>: jge 0xffffffff811f0240
<n_tty_read+1049>
0xffffffff811f0224 <n_tty_read+1021>: test %rdx,%rdx
0xffffffff811f0227 <n_tty_read+1024>: jle 0xffffffff811f0240
<n_tty_read+1049>
0xffffffff811f0229 <n_tty_read+1026>: mov -0xd8(%rbp),%eax
0xffffffff811f022f <n_tty_read+1032>: sub %cx,%ax
0xffffffff811f0232 <n_tty_read+1035>: add -0xb0(%rbp),%eax
0xffffffff811f0238 <n_tty_read+1041>: mov %ax,0x21e(%r13)
0xffffffff811f0240 <n_tty_read+1049>: mov %r13,%rdi
0xffffffff811f0243 <n_tty_read+1052>: callq 0xffffffff811f37f3
<tty_flush_to_ldisc>
0xffffffff811f0248 <n_tty_read+1057>: testb $0x10,0x21c(%r13)
0xffffffff811f0250 <n_tty_read+1065>: je 0xffffffff811f0261
<n_tty_read+1082>
0xffffffff811f0252 <n_tty_read+1067>: cmpl $0x0,0x478(%r13)
0xffffffff811f025a <n_tty_read+1075>: jne 0xffffffff811f026f
<n_tty_read+1096>
0xffffffff811f025c <n_tty_read+1077>: jmpq 0xffffffff811f0621
<n_tty_read+2042>
0xffffffff811f0261 <n_tty_read+1082>: cmpl $0x0,0x260(%r13)
0xffffffff811f0269 <n_tty_read+1090>: jle 0xffffffff811f0621
<n_tty_read+2042>
0xffffffff811f026f <n_tty_read+1096>: mov -0x110(%rbp),%rax
0xffffffff811f0276 <n_tty_read+1103>: movq $0x0,(%rax)
0xffffffff811f027d <n_tty_read+1110>: cmpl $0x0,-0x9c(%rbp)
0xffffffff811f0284 <n_tty_read+1117>: mov -0x40(%rbp),%rax
0xffffffff811f0288 <n_tty_read+1121>: je 0xffffffff811f0376
<n_tty_read+1359>
0xffffffff811f028e <n_tty_read+1127>: cmp -0xd8(%rbp),%rax
0xffffffff811f0295 <n_tty_read+1134>: jne 0xffffffff811f0376
<n_tty_read+1359>
0xffffffff811f029b <n_tty_read+1140>: jmpq 0xffffffff811f033b
<n_tty_read+1300>
0xffffffff811f02a0 <n_tty_read+1145>: mov -0xd0(%rbp),%rdi
0xffffffff811f02a7 <n_tty_read+1152>: callq 0xffffffff811eb980
<tty_hung_up_p>
0xffffffff811f02ac <n_tty_read+1157>: test %eax,%eax
0xffffffff811f02ae <n_tty_read+1159>: jne 0xffffffff811f052d
<n_tty_read+1798>
0xffffffff811f02b4 <n_tty_read+1165>: test %r15,%r15
0xffffffff811f02b7 <n_tty_read+1168>: je 0xffffffff811f052d
<n_tty_read+1798>
0xffffffff811f02bd <n_tty_read+1174>: mov -0xd0(%rbp),%rdx
0xffffffff811f02c4 <n_tty_read+1181>: testb $0x8,0x39(%rdx)
0xffffffff811f02c8 <n_tty_read+1185>: je 0xffffffff811f02da
<n_tty_read+1203>
0xffffffff811f02ca <n_tty_read+1187>: movq
$0xfffffffffffffff5,-0xa8(%rbp)
0xffffffff811f02d5 <n_tty_read+1198>: jmpq 0xffffffff811f052d
<n_tty_read+1798>
0xffffffff811f02da <n_tty_read+1203>: mov -0x108(%rbp),%rcx
0xffffffff811f02e1 <n_tty_read+1210>: mov 0x8(%rcx),%rax
0xffffffff811f02e5 <n_tty_read+1214>: testb $0x4,0x10(%rax)
0xffffffff811f02e9 <n_tty_read+1218>: je 0xffffffff811f02fb
<n_tty_read+1236>
0xffffffff811f02eb <n_tty_read+1220>: movq
$0xfffffffffffffe00,-0xa8(%rbp)
0xffffffff811f02f6 <n_tty_read+1231>: jmpq 0xffffffff811f052d
<n_tty_read+1798>
0xffffffff811f02fb <n_tty_read+1236>: mov $0xfff,%eax
0xffffffff811f0300 <n_tty_read+1241>: sub 0x260(%r13),%eax
0xffffffff811f0307 <n_tty_read+1248>: test %eax,%eax
0xffffffff811f0309 <n_tty_read+1250>: jg 0xffffffff811f0324
<n_tty_read+1277>
0xffffffff811f030b <n_tty_read+1252>: xor %eax,%eax
0xffffffff811f030d <n_tty_read+1254>: testb $0x10,0x21c(%r13)
0xffffffff811f0315 <n_tty_read+1262>: je 0xffffffff811f0324
<n_tty_read+1277>
0xffffffff811f0317 <n_tty_read+1264>: xor %eax,%eax
0xffffffff811f0319 <n_tty_read+1266>: cmpl $0x0,0x478(%r13)
0xffffffff811f0321 <n_tty_read+1274>: sete %al
0xffffffff811f0324 <n_tty_read+1277>: mov %r15,%rdi
0xffffffff811f0327 <n_tty_read+1280>: mov %eax,0xf0(%r13)
0xffffffff811f032e <n_tty_read+1287>: callq 0xffffffff812d5a02
<schedule_timeout>
0xffffffff811f0333 <n_tty_read+1292>: mov %rax,%r15
0xffffffff811f0336 <n_tty_read+1295>: jmpq 0xffffffff811f0522
<n_tty_read+1787>
0xffffffff811f033b <n_tty_read+1300>: mov -0xf0(%rbp),%rbx
0xffffffff811f0342 <n_tty_read+1307>: lea -0x31(%rbp),%rsi
0xffffffff811f0346 <n_tty_read+1311>: mov $0x1,%edx
0xffffffff811f034b <n_tty_read+1316>: mov %r13,%rdi
0xffffffff811f034e <n_tty_read+1319>: movb $0x0,-0x31(%rbp)
0xffffffff811f0352 <n_tty_read+1323>: mov %rbx,-0x40(%rbp)
0xffffffff811f0356 <n_tty_read+1327>: callq 0xffffffff812008ac
<tty_audit_add_data>
0xffffffff811f035b <n_tty_read+1332>: mov -0x31(%rbp),%al
0xffffffff811f035e <n_tty_read+1335>: mov -0xd8(%rbp),%rcx
0xffffffff811f0365 <n_tty_read+1342>: callq 0xffffffff811949a0
<__put_user_1>
0xffffffff811f036a <n_tty_read+1347>: test %eax,%eax
0xffffffff811f036c <n_tty_read+1349>: jne 0xffffffff811f043d
<n_tty_read+1558>
0xffffffff811f0372 <n_tty_read+1355>: decq -0x78(%rbp)
0xffffffff811f0376 <n_tty_read+1359>: testb $0x10,0x21c(%r13)
0xffffffff811f037e <n_tty_read+1367>: jne 0xffffffff811f0456
<n_tty_read+1583>
0xffffffff811f0384 <n_tty_read+1373>: jmpq 0xffffffff811f047a
<n_tty_read+1619>
0xffffffff811f0389 <n_tty_read+1378>: mov 0x25c(%r13),%eax
0xffffffff811f0390 <n_tty_read+1385>: mov -0x88(%rbp),%rbx
0xffffffff811f0397 <n_tty_read+1392>: lock btr %eax,(%rbx)
0xffffffff811f039b <n_tty_read+1396>: sbb %r14d,%r14d
0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx
0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax
0xffffffff811f03ac <n_tty_read+1413>: mov -0xb8(%rbp),%rdi
0xffffffff811f03b3 <n_tty_read+1420>: movsbl (%rax,%rdx,1),%ebx
Below is the output of bt -a command in crash
bt -a
PID: 0 TASK: ffffffff814204b0 CPU: 0 COMMAND: "swapper"
#0 [ffff880033007e80] crash_nmi_callback at ffffffff8101fbc9
#1 [ffff880033007e90] notifier_call_chain at ffffffff81065893
#2 [ffff880033007ed0] atomic_notifier_call_chain at ffffffff810658dd
#3 [ffff880033007ee0] notify_die at ffffffff8106597f
#4 [ffff880033007f10] do_nmi at ffffffff8100dc5d
#5 [ffff880033007f50] nmi at ffffffff812d76b0
[exception RIP: mwait_idle+163]
RIP: ffffffff81013029 RSP: ffffffff813e3eb8 RFLAGS: 00000246
RAX: 0000000000000000 RBX: ffffffff813e3fd8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffffffff813e3fd8 RDI: ffffffff81522308
RBP: ffffffff813e3ec8 R8: 0000000000000000 R9: ffff88003306e290
R10: 0000000000012d80 R11: 0000000000000000 R12: ffffffff8147e368
R13: ffffffff814ccb30 R14: ffffffff814cdfa0 R15: ffffffff813e3fa8
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
--- <NMI exception stack> ---
#6 [ffffffff813e3eb8] mwait_idle at ffffffff81013029
#7 [ffffffff813e3ed0] cpu_idle at ffffffff8100af21
PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"
#0 [ffff88031ce759d0] machine_kexec at ffffffff81024486
#1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230
#2 [ffff88031ce75b20] oops_end at ffffffff8100fa38
#3 [ffff88031ce75b50] no_context at ffffffff8102d801
#4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9
#5 [ffff88031ce75c70] bad_area at ffffffff8102da41
#6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19
#7 [ffff88031ce75cf0] page_fault at ffffffff812d7425
[exception RIP: n_tty_read+1420]
RIP: ffffffff811f03b3 RSP: ffff88031ce75da8 RFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044
RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c
RBP: ffff88031ce75eb8 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044
R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#8 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e
#9 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8
#10 [ffff88031ce75f40] sys_read at ffffffff810ebe48
#11 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2
RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212
RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000
RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000
RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
PID: 0 TASK: ffff88031e0e3540 CPU: 2 COMMAND: "swapper"
#0 [ffff880033047e80] crash_nmi_callback at ffffffff8101fbc9
#1 [ffff880033047e90] notifier_call_chain at ffffffff81065893
#2 [ffff880033047ed0] atomic_notifier_call_chain at ffffffff810658dd
#3 [ffff880033047ee0] notify_die at ffffffff8106597f
#4 [ffff880033047f10] do_nmi at ffffffff8100dc5d
#5 [ffff880033047f50] nmi at ffffffff812d76b0
[exception RIP: mwait_idle+163]
RIP: ffffffff81013029 RSP: ffff88031e0e5ef8 RFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff88031e0e5fd8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88031e0e5fd8 RDI: ffffffff81522308
RBP: ffff88031e0e5f08 R8: 0000000000000000 R9: ffff88003302e290
R10: 0000000000012d80 R11: 0000000000000000 R12: ffffffff8147e368
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
--- <NMI exception stack> ---
#6 [ffff88031e0e5ef8] mwait_idle at ffffffff81013029
#7 [ffff88031e0e5f10] cpu_idle at ffffffff8100af21
PID: 0 TASK: ffff88031e113580 CPU: 3 COMMAND: "swapper"
#0 [ffff880033067e80] crash_nmi_callback at ffffffff8101fbc9
#1 [ffff880033067e90] notifier_call_chain at ffffffff81065893
#2 [ffff880033067ed0] atomic_notifier_call_chain at ffffffff810658dd
#3 [ffff880033067ee0] notify_die at ffffffff8106597f
#4 [ffff880033067f10] do_nmi at ffffffff8100dc5d
#5 [ffff880033067f50] nmi at ffffffff812d76b0
[exception RIP: mwait_idle+163]
RIP: ffffffff81013029 RSP: ffff88031e115ef8 RFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff88031e115fd8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88031e115fd8 RDI: ffffffff81522308
RBP: ffff88031e115f08 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000800 R11: 0000000000000000 R12: ffffffff8147e368
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
--- <NMI exception stack> ---
#6 [ffff88031e115ef8] mwait_idle at ffffffff81013029
#7 [ffff88031e115f10] cpu_idle at ffffffff8100af21
Please let me know if you need any other details.
Thanks and Regards
Shashidhara
-----Original Message-----
Sent: Thursday, June 23, 2011 9:35 PM
To: Discussion list for crash utility usage,maintenance and
development
Subject: Re: [Crash-utility] Unable to switch stack frames while
using
crash
----- Original Message -----

Post by Dave Anderson
BTW, are you sure about that?
Presuming that the "tty" pointer is ffff8802cbd54800 as you've
shown

below,

Post by Dave Anderson
and therefore tty->read_buf is 0xffff8802cbfe6000 and
tty->read_tail

is 0,

Post by Dave Anderson
then the statement above would be simply be reading
tty->read_buf[0],

or

Post by Dave Anderson
virtual address 0xffff8802cbfe6000. But the oops shows it faulting
on

a

Post by Dave Anderson
BUG: unable to handle kernel NULL pointer dereference at

0000000000000005
Just for my own sanity, can you either attach the
"drivers/char/n_tty.c"
from *your* specific kernel, or get the source-code/line-number data
from
the embedded gdb module?
If you don't have the n_tty.c file readily available, you can get
the
source-code/line-number data of a particular function by doing
something
Get the line number of the beginning of n_tty_read(), which in my
kernel
crash> gdb list n_tty_read
1695 * This code must be sure never to sleep through a hangup.
1696 */
1697
1698 static ssize_t n_tty_read(struct tty_struct *tty, struct file
*file,
1699 unsigned char __user *buf, size_t nr)
1700 {
1701 unsigned char __user *b = buf;
1702 DECLARE_WAITQUEUE(wait, current);
1703 int c;
1704 int minimum, time;
crash>
Then get the line number of the next function in the file, which is
crash> gdb list n_tty_write
1918 * lock themselves)
1919 */
1920
1921 static ssize_t n_tty_write(struct tty_struct *tty, struct file
*file,
1922 const unsigned char *buf, size_t nr)
1923 {
1924 const unsigned char *b = buf;
1925 DECLARE_WAITQUEUE(wait, current);
1926 int c;
1927 ssize_t retval = 0;
And then dump the whole n_tty_read() function (plus some extra
crash> gdb list 1698,1920
1698 static ssize_t n_tty_read(struct tty_struct *tty, struct file
*file,
1699 unsigned char __user *buf, size_t nr)
1700 {
1701 unsigned char __user *b = buf;
1702 DECLARE_WAITQUEUE(wait, current);
1703 int c;
1704 int minimum, time;
1705 ssize_t retval = 0;
1706 ssize_t size;
1707 long timeout;
1708 unsigned long flags;
1709 int packet;
1710
1712
1713 BUG_ON(!tty->read_buf);
1714
1715 c = job_control(tty, file);
1716 if (c < 0)
1717 return c;
1718
1719 minimum = time = 0;
1720 timeout = MAX_SCHEDULE_TIMEOUT;
1721 if (!tty->icanon) {
1722 time = (HZ / 10) * TIME_CHAR(tty);
1723 minimum = MIN_CHAR(tty);
...
And lastly, since the crash occurred at
IP: [<ffffffff811f03b3>] n_tty_read+0x58c/0x818
crash> dis -rl n_tty_read+0x58c
...
And then post all of that data.
Dave
--
Crash-utility mailing list
https://www.redhat.com/mailman/listinfo/crash-utility
Information transmitted by this e-mail is proprietary to MphasiS,
its
associated companies and/ or its customers and is intended
for use only by the individual or entity to which it is addressed,
and
may contain information that is privileged, confidential or
exempt from disclosure under applicable law. If you are not the
intended recipient or it appears that this mail has been forwarded
to you without proper authority, you are notified that any use or
dissemination of this information in any manner is strictly
prohibited. In such cases, please notify us immediately at
--
Crash-utility mailing list
https://www.redhat.com/mailman/listinfo/crash-utility

--
Crash-utility mailing list
https://www.redhat.com/mailman/listinfo/crash-utility
Information transmitted by this e-mail is proprietary to MphasiS, its
associated companies and/ or its customers and is intended
for use only by the individual or entity to which it is addressed, and
may contain information that is privileged, confidential or
exempt from disclosure under applicable law. If you are not the
intended recipient or it appears that this mail has been forwarded
to you without proper authority, you are notified that any use or
dissemination of this information in any manner is strictly
prohibited. In such cases, please notify us immediately at
--
Crash-utility mailing list
https://www.redhat.com/mailman/listinfo/crash-utility

[Andrew Suffield]

Post by Dave Anderson
So it just seems impossible to recreate even with an intervening
n_tty_close().

Yeah, I can't see a way to do it either. I guess that leaves hardware
failure. (memory/cpu in particular)